Student Charged $14k on Stolen Google Cloud Credentials

by nitins_jakta on 4/9/2019, 12:58:13 AM with 35 comments
Hi,

In 2017, I made a Google Cloud Account to use Google Maps API for a Computer Science student group project and put my debit card in. I naively put a $5 account notification in, thinking it was a cap. This project was defunct after 2017 and I should have just closed the Cloud account.

All was fine up until January 2019 when the Google Cloud Credentials were somehow stolen and over the course of two days on Google Maps API, racked up enough API calls to generate over $14k invoice. I disabled the Google Cloud Account a day after I noticed an email from Google Cloud. Google Cloud did try to use debit card to deduct from checking account, but I don't leave thousands sitting around in it, so charge was declined.

I talked to Google Cloud Billing and they have not been helpful, telling me to contact my bank. Today, I got a scary email from a collections agency demanding I login to my Google Cloud account and pay the bill! Worst part is, this API used to be free, until Google started charging exorbitant amounts for it.

I know I did not make these API calls -- if you looked at the call volume history, there was nothing for well over a year, until those two days in 2019, it started going crazy (and the project is not running on any server or being used in any way). I suspect a group member might have accidentally leaked the credentials.

I know AWS has waived costs[1] like this in the past, but Google is not known for customer support. I should have been more proactive in setting up a cap.

Appreciate any advice or Google contacts to talk to an actual human. Should I see if Google is willing to actually verify this was unauthorized usage or just lower the bill? I'll eat a few thousand just to make this go away.

To say GCP has left a sour taste in my mouth is an understatement!

Thanks for reading.

[1] https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

  • by boulos on 4/9/2019, 5:04:02 AM

    Sorry this happened to you! Feel free to send me your case number (email in profile), and I'll escalate it.

    The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.

  • by scarface74 on 4/9/2019, 2:53:44 AM

    I’ve heard so many stories of something similar happening on AWS and after an email to support, all of the charges were dropped.

    This isn’t exactly helping Google to fight the narrative that it isn’t good with customer support and they can’t be trusted as a platform for business.

    So if you were a person deciding who to choose as your cloud provider, who are you going to choose?

    AWS - “No one ever got fired for choosing AWS”

    Microsoft - well known for their enterprise support and there are plenty of MS Shops out there.

    Or

    Google?

  • by segmondy on 4/9/2019, 3:03:44 AM

    While you're figuring this out, backup all your data on Google. Google is crazy and could possibly delete all your accounts and data.

  • by unknownkadath on 4/9/2019, 2:51:55 AM

    Before disputing the charge, be sure to back up all data and contact info from your Google accounts. Fighting charges has been known to trigger account lockouts with no appeal.

  • by applecrazy on 4/9/2019, 3:59:37 AM

    Did you check your Github repos and associated commit history for accidental push of secret files? There's an article on the HN front page describing secret leakage in Github repos (the most common is Google API keys, go figure)[1]. I imagine somebody out there has a bot to monitor pushes in realtime to extract secrets. You or a team member might have leaked keys in a similar manner.

    [1]: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...

  • by londons_explore on 4/9/2019, 3:18:44 AM

    Google will typically waive charges in cases like this.

    The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.

  • by foobarbazetc on 4/9/2019, 1:02:52 AM

    Make them prove you used it to generate the charges. Make them provide IPs etc.

    You need to say it was used fraudulently and you don’t agree to the charges.

  • by samfisher83 on 4/9/2019, 4:21:37 AM

    Next use a credit card. Basically thanks to credit card laws the bank will go tell google to f off and give you your money back. Debit cards don't have the same protection, but just call your bank or OCS (https://www.occ.treas.gov/). They have a little more bite.

  • by cjbprime on 4/9/2019, 3:16:02 AM

    Keep trying to talk to them and explain -- so far every instance like this I've heard about was refunded. Good luck.

  • by kkarakk on 4/9/2019, 12:42:55 PM

    I dunno if google lets you do this but amazon/azure will pretty reliably let you create new free tier accounts with fake emails and access them from the same IP. i just create a new debit/credit card every 6 months(it's pretty hassle free in india).

    i do pay for production instances, i just don't want to mess around on those production instances