When I Woke Up To ₹ 28,000 Bill From Google Cloud Not Knowing Why
A story about the morning when I woke up to find that I was charged ₹ 28,000 by Google Cloud, with no prior memory of having used anything on Google C
7 min read
Originally posted on my website
Until recently, like many of us, the first thing I used to do in the morning would be to check my mobile phone. I realised that it was taking about 20 mins to half an hour of my morning time. So I stopped doing it and replaced it with keeping my eyes closed and meditating instead for 15-20 mins. But the incident I'm going to tell you about happened when I still used to look up for my mobile phone first thing in the morning, in half asleep state, with one eye open and the other closed, hid behind the pillow contour.
On 3rd September, 2020, I woke up to an automated email from Google Cloud Platform. First response – shock. It was a bill of $377. That amount meant a lot because I didn't earn much in INR, and it was a significant portion of my salary. Without even opening my Google Cloud console, I started a chat with Google Cloud support. In about a couple of hours, they told me in detail what I was billed for.
I worked for Goibibo at the time. Goibibo is one of the largest online travel agencies in India. Two months before this happened, I was working on the hotels details page's Google Maps project — you know — to show the location of the hotel in the map, and all the points of interest around that hotel, and the ability to search for any location in reference to the hotel.
I read a partiular line in one of the Google Cloud support communication emails:
Just a quick disclaimer, we cannot reveal domain information if the usage is from a domain outside of your authorized ones due to our privacy and security policy.
And then it hit me.
Goibibo's google cloud account didn't have
/* Only use this key for production, comment the one below */ // export const googleMapsApiKey = `<goibibo account's api key>`; /* * WARNING: Do not push the key to production. Is only meant * for development purposes only */ export const googleMapsApiKey = `<my personal account's api key>`;
I frantically checked the code in the main branch to see if I had accidetally forgotten to follow my own advice. That was in fact the case, and I did a big 🤦🏽♂️
I git blamed the file to find the pull request which had enabled this blunder. Then I sent its screenshot to both the google cloud support, apologizing for an honest mistaken, and to my team's slack channel to blare loudly at them for not having done a decent code review.
I had also forgotten to restrict my personal account's API key to allow requests to Google Maps APIs from only
That meant that
Had someone extracted that API key by watching the API request from browser developer tools, they could have potentially bombarded requests to Google Maps APIs billed against my personal account from any domain.
I didn't detect this blunder for a month, because for the first month, google had used the $100 free credit that I had gotten from somewhere. So in the first month's billed it showed
$0. It was only a month later that that month's $277 showed as amount charged on my credit card.
On multiple requests the Google Cloud team refunded that amount after taking confirmations from me that I had read and now understood all the ways to restrict access of api keys, and to set billing alerts on google cloud projects. After giving multiple assurances and acknowledgements that I had indeed read up all the docs that taught how to do all these things, I found peace. 🕊✌🏼☮️