back up as of now (10:10a PDT)

This sure makes it easy to know who is hosted by google by going to downdetector.com.

Etsy is 404 too: https://www.etsy.com

Seems to be a bigger issue.

edit: Nest is down too: http://nest.com

Fitbit.com is 404 too: https://www.fitbit.com

Big GCP issue?

edit2: Downdetector.com shows multiple website and services as down, including Pokemon GO or Rocket League.

GCP status page is still green all over the board: https://status.cloud.google.com

19:10 CET update: Some websites are coming back, including spotify.com, but their app still does not work for me.

information about outage just added to GCP status page, direct link: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-11-16 09:53 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-11-16 10:40 US/Pacific with current details.

We apologize to all who are affected by the disruption.

19:20 CET update:

Description: We believe the issue with Cloud Networking is partially resolved.

Customers will be unable to apply changes to their load balancers until the issue is fully resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2021-11-16 11:28 US/Pacific with current details.

Spotify desktop app still not working for me.

19:45 CET: Spotify app is back online for me.

Global: Experiencing Issue with Cloud networking

Incident began at 2021-11-16 10:10 (all times are US/Pacific).

https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

Looks like perhaps an issue with Google Load Balancer. We have a load balancer in front of Google Storage Buckets, and can access resources directly from the buckets, but getting 404 when going through the load balancer.

Half the internet is down because of a Google Cloud global issue on their load balancers, including Spotify and Etsy and GCP status is all green: https://status.cloud.google.com If you ever wondered why GCP is a distant third runner in the enterprise cloud space, here is your answer.

Ironically it appears that IsItDownRightNow? is also down, although that could because they're experiencing what is basically going to be the equivalent of a DDOS.

Time to switch to https://bandcamp.com.

I wonder if a legal discovery will ever find internal status dashboards that reflect reality rather than fictitious SLA liability-aware status pages.

This is clearly the hottest thing on HN right now, and it was bumped from #1 to #6, anyone knows why? Is it some kind of bot protection?

https://linear.app/ is also down

We found extra rules in our GCLB routing config - removing them restored our service.

Netlify is also failing for us, and reporting bad TLS certs. Not sure if they use GCP https://www.netlifystatus.com/

We bypassed our Google load balancer and pointed DNS directly at the IP addresses of our servers and that seems to have helped

The title of the 404 page on all the down sites has an extra "1" after the exclamation points: "Error 404 (Not Found)!!1"

Sites that are down according to https://downdetector.com include Spotify, Discord, Snapchat, Etsy, Pokemon Go, Epic Games, Target, Paramount+, Evernote

Global: Experiencing Issue with Cloud networking Incident began at 2021-11-16 10:10 (all times are US/Pacific). https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

GCP outage? Status page shows green but a bunch of sites seem down (Rocket League most importantly).

I wondered why our alerts started going nuts. Seems like basically every global Google Cloud load balancer went down. Doesn't seem to affect single-region network load balancers.

Edit: All of ours are back up. Some other services still seem down though.

Funny thing is when you google Home Depot or Paramount Plus you get ads served by Google as the first result. When click on it Google then shows you a 404 page. I wonder if they'll get a refund on their Adwords campaign.

I also wonder how many companies didn't want to admit they were using Google for their infrastructure. Downdetector shows AWS being affected, it'd be embarrassing if they were caught using Google Cloud Platform.

ok, so our API is down. We're on GCP...

https://api.newscatcherapi.com/v2/search

See also https://news.ycombinator.com/item?id=29243740

We were down. Just came back. Things seem to be resolving.

https://www.navidrome.org/

Self hosted Spotify. Compatible with subsonic clients.

My company was seeing GCP Airflow environments not responding, but they seem to have recovered in the past few minutes.

https://overleaf.com/ is also 404 now

https://toggl.com/ is also affected.

I have a few services running from the same GKE cluster, same ingress controller, same nodes, same GLB, same everything.

Some are 404ing at the moment and others work just fine. Feels like a GLB issue.

Nothing in my GCP dashboard seems to be aware of the issue however.

Only reason I found out is because I use an external service to ping me if a site is down.

https://www.windy.com down, same issue.

The 404's changed into 502's.. I guess that's progress. Fingers crossed it's back up soon

Yes, but we're still on DEFCON 5.

Seeing the same - I have projects in us-east1 that went offline first, then us-west1 went offline a few minutes after. Everything green on their status page and nothing in the dashboard - everything returns a 404 so I'm assuming a really high level LB just took a dump.

I don’t understand why people post website outages.

Do you think the DevOps teams at these billion dollar streaming companies are so clueless that they don’t have monitoring in place?

Do you think that people who go to a site when it’s down don’t see the same thing?

So whose awareness does this serve?

Looks like it's made it to the Google status page: https://status.cloud.google.com/incidents/6PM5mNd43NbMqjCZ5R...

Is it regional? Surprised Spotify is not active-active on other platforms like AWS and Azure.

Right now homedepot.com and the APIs that drive their mobile app are down too.

Not Google's best week.

Might be Google Cloud outage: https://news.ycombinator.com/item?id=29243753

Everything seems to be working on our end as of 1:08 PM EST.

Affecting us. Busiest time of the year and now down 20 min. It's the Global Load Balancer, so god knows what bit of the global edge has been taken out.

One of the websites I've noticed this on is back up.

- discord doesn't allow me to connect either.

1:10PM - either it has resolved itself, or a regional issue, but I'm not seeing anything being down from the East coast USA.

As far as I can tell everything is up it's just that our load balancers aren't routing traffic and just returning 404s.

Funny enough, datadog, which I was using to investigate on of my vercel sites, is down too

Yeah, Vercel is running some GCP services it seems

I knew Google would one day take control of the web, just thought they'd have a more clever way of doing it.

Oh okay, thought something was on my end.

Rocket league perhaps epic games even

I was experienced this issue with my spotify app. Initially thought, it was my internet issue lol.

Seeing this across the board with providers on GCP. Firestore however does not appear to be down

https://downdetector.com/

yup

That's why I like the clouds.

Discord down

My stuff is running OK on GCP, with GKE usage. Maybe it's related to nameservers?

It looks like everything is back now. That was a short outage by recent standards...

Seems like another DNS issue - switching to Cloudflare 1.1.1.1 got me back online...

Seems to be an issue with GOOGLE cloud load balancer. our website is down too

Seems to be coming back up.

Also experience that with all my GCP related infrastructure (Europe)

Our infra in us-central1 behind gcp lb is impacted but not us-west1

My app was down too. Can confirm it is most likely the GCloud LB

Our instances started working again, so seems to be fixed

Pokémon GO also down

Same here, seems to be an issue with load balancers

GCP - my Cloud Run containers are giving 500's

overleaf.com also

CBS.com is down, and some NYT pages as well.

Snapchat also having issues refreshing

I noticed Discord went down for me.

veed.io is down too! Same problem

it was back in a few minutes. But that wasw pretty weird. App Engine was affected.

Things are back online again!

Lowes search was also down.

Discord seems to be down.

We're also affected.

Seems to be back already

Discord is down too.

Seems to be fixed.

It's back up

It's back!

looks like it's back up?

nice