Disclosure: I work on Google Cloud (but disclaimer, I'm on vacation and so not much use to you!).

We're having what appears to be a serious networking outage. It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages.

There are backup plans, of course, but I wanted to at least come here to say: you're not crazy, nothing is lost (to those concerns downthread), but there is serious packet loss at the least. You'll have to wait for someone actually involved in the incident to say more.

Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.

https://www.reddit.com/r/sysadmin/comments/bw1gye/most_googl...

https://www.zdnet.com/article/some-internet-outages-predicte... 768k Day

There goes 3 nines for June and for Q2. I guess everyone gets a 10% discount for the month? https://cloud.google.com/compute/sla

You know this reminds me of a bad taste that Google Sales team left when I asked for some of my billing that I was unaware of running after following a quickstart guide.

AWS refunded me in the first reply on the same day!

GCP sales rep just copy pasted a link to a self support survey that essentially told me, after a series of YES or NO questions that they can't refund me.

So why not just tell your customers like it is? Google Cloud is super strict when it comes to billing. I have called my bank to do a chargeback and put a hold on all future billing with GCP.

I'm now back to AWS and still on a Free Tier. Apparently the $300 Trial with Google Cloud did not include some critical products, AWS Free tier makes it super clear and even still I sometimes leave something running on and discover it in my invoice....

I've yet to receive a reply from Google and its been a week now.

I do appreciate other products such as Firebase but honestly for infrastructure and for future integration with enterprise customers I feel AWS is more appropriate and mature.

GCP status page is worthless as it's always happy and green when production systems are down and then they might acknowledge something an hour later

Google Cloud is the number 4 most monitored status page on StatusGator and Google Apps is number 12. In addition, at least 20 other services we monitor seemingly depend on Google Cloud because they all posted issues as soon as Google went down.

It's always interesting to see these outages at large cloud providers spider out across the rest of the internet, a lot of the world depends on Google to stay up.

And thus was ruined hundreds or thousands of pleasant Sunday afternoons.

I don’t miss being on pager duty one bit. I see it looming in my headlights, sadly.

That feeling when you open https://console.cloud.google.com and see that you don't have your Kubernetes clusters and CloudSQL databases, but CTA to create first.

Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?

This is your yearly reminder to resist centralization of the internet.

It seems the AdWords anti-spam system is down, which means anyone can put a billion dollar bid on every keyword and get their ads showing on every Google search for every query.

Systems that fail 'open'...

Level(3) one of the biggest backbones has issues too, might be related

https://downdetector.com/status/level3

Funny how as soon as I realized that Gmail and Google Sheet aren’t working properly I rushed to HN to figure out what’s going on. I love this community!

The two Google Cloud networking incidents are:

Incident #19008 began at 2019-06-02 12:48. https://status.cloud.google.com/incident/cloud-networking/19...

Incident #19009 began at 2019-06-02 12:53. https://status.cloud.google.com/incident/cloud-networking/19...

Times are US/Pacific

I also noticed that Google search stop indexing news articles.

So I searched for "gmail down" on bing and I got some results [1]. But searching on Google for "gmail down" does not return any results [2].

[1] https://www.bing.com/news/search?q=gmail+down&qs=n&form=QBNT

[2] https://www.google.com/search?q=gmail+down&source=lnms&tbm=n...

And Gmail too doesn't feel very well today.

  [21:55:19] POP< +OK send PASS
  [21:55:19] POP> PASS ********
  [21:55:21] POP< +OK Welcome.
  [21:55:21] POP> STAT
  [21:55:21] POP< -ERR [SYS/TEMP] Temporary system problem.
  Please try again later.

Anyone using both AWS and GCP that can form an opinion on availability of both? As a GCP customer I am not very happy with theirs.

Nice overview : https://downdetector.com/

I was playing around this afternoon with appengine, and thought I broke one my projects when I started getting 502 back.

There appears to be some irregularities on consumer services as well that are of course certainly related, youtube was behaving a bit oddly for me.

The impact seems to be cascading down from just GCE to other services as well - that status page certainly does not reflect the reality of the situation. You can't even sign into GCP right now, and things that run on GCE, like appengine seem impacted.

Nest is down for me right now.

It's amazing how far-reaching outages can be these days.

Yep, I can no longer see my Cloud SQL database - it's as if I've never created one at all. Really hoping this is just an issue displaying it and that Google hasn't punted my infrastructure and backups.

When talking about GCE being down please also mention what regions you are talking about

Other google services are also affected:

https://www.google.com/appsstatus#hl=en&v=status

https://i.imgur.com/pcqwwA4.png

Phase 2 of mitigation was completed a few minutes ago and GCE is expecting improvement shortly.

It seems crazy to me that Google Cloud can have this level of instability but I, on the other hand, can never remember google.com going down.

Why are they operating one with a different networking infrastructure from the other?

Right now this is at the 'good at least it's not mine to fix or worry about' kind of like 'and the reason I choose IBM' [1]. I can just sit back and wait for gmail to work correctly. Now at the point it starts to last what I would consider a long time well then I will have things to worry about.

One thing with gmail though. When it's down it's similar to a snow storm if you only do business in a city. Everyone is impacted and everyone understands a missed deadline is unavoidable.

[1] For those not old enough to know what I mean read this: https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalc...

Looks like my Gmail is back, but I don't have any emails from while it was down. Yikes.

Edit: just got one email from the downtime, so perhaps my initial conclusion was incorrect

So just got a new phone and trying to resync my 2FA for AWS but I can't because Gmail is down. Ffs.

Just 2 weeks after I migrated a DB cluster from Azure to Google Cloud thinking things would get better.

We're hosting an open global Zoom call for all engineers affected by the outage, join us at https://zoom.us/j/793450725

Does anyone know if this is a regional or global outage?

I can see my GKE clusters in one region but not in another, so I am guessing it's the former.

Looks like we'll need a cluster in each region going forward...

I tried multiple times to setup a Google Wifi router today. Wifi would work but the app said it was offline. Perhaps I am not insane or incompetent after all

https://status.cloud.google.com/incident/compute/19003 Does status update mean only the status. Won't there be words like "sorry", "apologize" & "inconvenience"! Only PR is responsible for those words?

The status page took a while to show issues. My app was down, and Twitter knew google cloud was down before the official status page.

Its funny how searching for "Google Down" on Google and filtering for results within 1 hour yields nothing.

Is shopify on google cloud? i noticed they are having issues too

Not sure if related, but I was going to a BBQ yesterday and myself and 3 other people got lost because Google Maps app glitched out, directing us to the wrong places. If you search twitter for #googlemaps tons of people have the same issue. Surprised no one has posted about it.

https://status.cloud.google.com/incident/compute/19003

>We will provide an update at 16:00 US/Pacific.

it's 16:22 and no updates were posted. a bit unprofessional..

Related:

https://news.ycombinator.com/item?id=20077275

https://news.ycombinator.com/item?id=20077293

0 issues at compute, reporting for europe-west3-b,

I appears that logging into the webmail solves at least POP mail problems. I tried my mail client and failed, then attempted a login to the webmail which worked. Gmail then asked me to confirm my recovery address and cellphone, which I did, and finally loaded the inbox page. I immediately attempted a connection through the POP client and this time it worked.

It might be something security related if it triggers a mandatory identity confirmation.

edit: I tried to send me a mail from another account and it worked but out of 4 or 5 mail checks at least two failed giving the same error.

[23:44:27] POP< -ERR [SYS/TEMP] Temporary system problem. Please try again later.

The problem seems much more complex.

The GCE console also affected, couldn't send a support ticket just getting errors.

That explains why my google home thing thinks it’s sub-zero even though I’m warm in shorts and a t-shirt.

https://pasteboard.co/IhBsyrsO.jpg

The last big outage, iirc, was Google didn't test their rollback procedures for router upgrades. I'll be very interested to hear if it's yet another change control problem that caused this outage.

I was finishing a university assignment with the deadline 90mins away.

I wanted to upload a video of the project to YouTube and add a link to it in the report. YouTube takes a long time to process the video, and then says it's unavailable.

I go to Vimeo: it's down.

I upload the video to Dropbox, and copy its link to the report.

But my report was a Google doc. And when I tried to export it as PDF (which I had not done yet) it couldn't do it. I never hated google more.

Eventually the video went through to YouTube, and I could export the PDF on the third try, but this really made me conscious of my dependance on Google.

Everything looking normal on our GKE / CloudSQL stuff (eu-west1)

Confirming issues on our end. I'm able to load up my console but when I go to Kubernetes Engine, I don't see my clusters. I'm monitoring closely on twitter

Can't wait for the postmortem!

Was certainly an interesting alert when my Cloud Functions started reporting downtime. Among the many things that dip in and out on what seems like a monthly basis, I’ve not seen them just drop out in quite a while. Hopefully they get things sorted out. I can’t really imagine what it looks like internally when this level of outage is going on, but I want to think everyone is fairly collected

Once again HN proves to be the best status monitor.

This might be all in my head but I've been experiencing really bad latency for like an hour, while browsing, and then I read this.

https://downdetector.com/

Pretty much every service is down

My builds are failing because it cannot download the chromium.

> Error: Download failed: server returned code 502. URL: https://storage.googleapis.com/chromium-browser-snapshots/Li...

And Thunderbird suddenly kept throwing prompts for my Gmail passwords, even after I sign back in. I hope it's related.

Google Colaboratory is back. At least I can access my github's notebooks and public notebooks from google drive.

G suite was broken for me 20 minutes ago (in Europe) but is working now. Perhaps things are starting to come back?

Github contribution graphs are also gone

Does Google also have some sort of listing on which consumer apps are particularly affected (i.e. Gmail, Hangouts, Docs, Sheets, etc).

The cloud components may be directly affected but for consumers, there's nothing which will provide info on what consumer facing services are getting some issues.

> We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

I'm not seeing anything at 12:47.

Majority of G Suite services are suffering service outages: https://www.google.com/appsstatus#hl=en&v=status

Keep on centralizing the internet with your stupid clouds and this is what happens.

Anyone experiencing issues with GCS? Seems highly intermittent and dependent on the location the request is coming from (maybe that's because it's a networking bug).

The status page says GCS is fine but that's highly unlikely.

It seems that it`s focused on the database stuff like firebase and firestore.

Took me a while to track latency issues to GCP. Wasn't expecting it. This also seems to affect some GAE instances and some of their products like google photos. At least according to my observations

I just can't reach google apps on my HTC m9 since yesterday. I am in West Africa. My Whatsapp crashed too and I lost all my previous threads. Is my issue related to Cloud being down?

Playing "is google on this website or not?" just became so much easier, simply see if website works or not.

Scary stuff. What happens when Murphy's law decides to crash things even more?

Yeah I was having trouble accessing my Gsuite apps, had a couple of 502s, which led me to check HN. While it doesn't give me 502 now, it's abnormally slow.

Was trying to set up SSL on a GKE cluster today. Guess I'll have to wait for tomorrow if I want to be able to tell my mistakes apart from Google's.

u.s. west: all our cloud compute is inaccessible rn.... our API is down, can't ssh into the servers, and also can't see them on the dashboard.

This must be why GitLab is giving me shit. They recently made the switch. Wonder if there is any second guessing going on over there right now.

Wondered why Snapchat was being weird today. Thought it was my pi-hole setup blocking something from working, but nope, it's Google!

In India, I could access Youtube, Gmail (web) and Google cloud console and GKE and Compute Engine instances in south-east asia region.

Looks like only GCE is down according to the status page now. I'm able to access my console for instances and GKE clusters.

btw, Google Analytics realtime is down as well.

Weird for Twitter to still be up and fully functioning. I thought they migrated everything to GCP this/last year?

You look up. A single ray of light has made its way to the earth. Some day, you hope, the sky will be clear again.

google services also f´ed up here in middle europe; cannot reach anything google related from hamburg, germany.

https://tour.golang.org is also down

Seems to have coincided almost exactly with my Chromecast stopping displaying my photos (in ambient mode).

Ah, wonder if this is affecting Google's SSO. It was super slow when I was trying it just now.

No wonder I was trying Webcodesk right now and it's not working it's all firebase, yay!

So that's why YouTube was being weird. I thought it was an extension problem or something.

We are on region us-east1 and our systems are still up. Specifically, we are on us-east1-b.

youtube streaming is also down

I think nobody mentioned but Adsense also is not updating for couple of hours.

I've noticed problems on GDrive (GSuite) and YouTube as well. Connected?

Couldn't load the support console to "me too" this one either!

There go the nine nines for this year? Is it more like four nines or so now?

Anything to do with China?

Hug your on-call engineer.

With Google Cloud incidents, most of the time whole regions fail, and with AWS generally only a region fails. Of course there would be exceptions, but Google Cloud does not make me feel safe as an outsider (and a user of multi-region AWS)

These things happen. That's OK. Here's what's not OK:

> We are investigating an issue with Google Compute Engine. We will provide more information by Sunday, 2019-06-02 12:45 US/Pacific.

The next update is at 12:59. Just ... no.

Working here but slow (I'm based in Central Canada).

So far the Ko list:

GCE, GKE, BQ, Pub/Sub, GAE

asia-south1 us-west1 us-central1 us-west2

Looks like Google Analytics isnt reporting stats either?

Snapchat is fixed but snap maps is still disabled

gitlab is slow too

Google Play is also experiencing massive issues.

I cant see any gke cluster in Brazil, or any VM.

A demo of “too big to fail [via antitrust]”?

Snapchat is back up but snap maps is down

So is Youtube.

could this be the result of another BGP hack ? cyberwarfare ? I am just speculating here big time.

GCP has been down since 11:50am and they acked it 35 mins later. They're great at leaving their customers in the dark.

gmail is down in Australia...

gmail also down/super slow atm for me (East Coast, USA)

Let's see if perfect leetcode skills will save the day. /s

vimeo.com is down.

Huawei just flexing its muscles. Nothing to see here, move along.

No, not really.

Ironically, I moved all of my objects off GCS today.

There is a loop in the spanning tree.....

forbes.com is down also?

My gmail it's down!

seems like only us down

Prediction: the final postmortem will say "someone pushed a bad config", just like most of the previous postmortems (and most of the internal postmortems as well, for Borg-based services). This is the cause of most other outages in other cloud providers as well. A really hard to solve problem.

So that's why I can't login to YouTube this morning...

just had rolld20 in the USA blow out a game I wonder if it is effected

On days like these I’m glad I don’t use any of the affected services

What I've realized from this: Google doesn't have an official status page for GCP. There are a few unofficial ones, but nothing official that I could find.