Google Cloud Suspension: How a Single Incident Disrupted a 100k-User Live App and 1 Million Photos
Overview
A live application serving roughly 100,000 users, with over a million customer photos and about $1 million in annual recurring revenue, recently experienced a near-total shutdown due to a Google Cloud suspension. The incident highlighted how a vulnerability in API design can cascade into widespread operational and data access problems for an entire business.
What happened
– The app relies on Google Maps and follows Google’s guidance to ship an API key inside the mobile application. Google’s own documentation notes that these keys are not secrets.
– The user discovered that if Gemini is enabled within the same Google Cloud project, the Maps API key could potentially be used to authenticate Gemini requests as well.
– An unauthorized party extracted the Maps key from the app (in the standard, documented location) and used it to perform Gemini calls, resulting in thousands of dollars in charges (approximately $4,200).
– The account owner had not knowingly used Gemini, had not signed up for it, and was unaware that the key could grant access to Gemini.
– There were supposed spending limits, but Google apparently auto-raised the billing tier at some point, allowing continuing charges.
Escalation and impact
– Google suspended the entire project for “abusive activity consistent with hijacking.”
– The consequence: immediate loss of access to all project resources, including the app, APIs, and user data (over a million photos tied to 100k users).
– When the suspension occurred, users could not access their photos, the administrator lost console access, key rotation was blocked, data could not be moved, and incident response was severely hampered.
– The only recourse available at that moment was to submit an appeal and wait for a response.
Key takeaways
– A single Google Cloud suspension can simultaneously disrupt the entire application, its APIs, and access to user data.
– Relying on a single project boundary means that business continuity is tightly coupled to the health and policy enforcement of one provider.
– Even when the underlying vulnerability is not the organization’s fault, the provider’s response can have outsized operational consequences.
What this means for developers and teams
– Understand the limits and consequences of tying multiple services to a single project. Consider isolation strategies that minimize cross-service impact.
– Review key management practices and consider how API keys are distributed and what access they grant, especially when related services share a project.
– Be aware of how automatic policy changes (e.g., billing tier shifts) can escalate costs unexpectedly and ensure proactive controls are in place.
– Establish robust incident response plans that include rapid access to support channels and documented appeals processes with cloud providers.
Current status and lessons
– The situation underscores the importance of proactive architecture choices, such as service isolation, separate environments for sensitive integrations, and clear data access boundaries.
– It also raises questions about provider guarantees for access to data and services during suspensions or policy enforcement events.
– The ongoing support process remains in progress, with the organization awaiting human review and guidance from Google.
Final note
This incident serves as a cautionary tale for any business relying on cloud-provider services: a misalignment between API design, key usage, and policy enforcement can lead to sudden, company-wide outages and data access losses. Continuous review of security practices, data governance, and contingency planning is essential to resilience in modern cloud-native applications.
