Gemini: How database hotspotting and a one-minute cache TTL amplified a major outage

Company and product

Google Gemini is Google’s AI assistant platform, available through web and mobile applications as well as integrations such as Gemini in Chrome. 

‍

Gemini relies on internal services to retrieve and manage tool-deployment metadata. During this incident, an issue in this underlying metadata path  disrupted prompt handling and associated conversational features across multiple Gemini surfaces.

What happened

On June 10, a spike in frontend Queries Per Second (QPS) pushed Gemini’s database system beyond its capacity threshold. The service was already operating near maximum utilization. This additional request volume triggered extreme read contention within the backend database responsible for managing tool-deployment metadata.

‍

The primary cause was an index design issue in the database. A column tracking deployment expirations contained a high volume of similar values, including specific metadata fields containing empty values.  These values clustered tightly on a small number of database shards, creating a "hot shard" condition where a fraction of the database absorbed a disproportionate share of the incoming traffic.

‍

An in-memory cache with a TTL of only one minute severely amplified the problem. Because cached tool metadata expired rapidly, the system constantly refreshed the database. As request volume increased, the internal tool-management service experienced a more than 10x surge in database calls because the cache hit rate dropped significantly. The storage layer then reached its capacity for handling incoming queries, with database failure rates rising to 60%. This led the cache hit rate to fall to 50%. 

Timeline

• June 10, 10:30 UTC  : User-impact window begins. Gemini users start experiencing elevated prompt failures across web, mobile, and Chrome integrations.
• During the incident: Google’s real-time monitoring detects high processing error rates. Engineering teams begin investigating database contention.
• During mitigation: Engineers apply Rate Access Control Lists (RateACLs) to the most affected database shards, enabling internal caches to recover.
• During mitigation: The team redistributes common database index values across a broader range to balance the load evenly across shards.
• During mitigation: Google increases the in-memory cache TTL from 1 minute to 20 minutes, significantly reducing database load. 
• During mitigation: Engineers shorten RPC deadlines for database queries to prevent request pile-ups during high latency.
• June 10 17:25 UTC: The elevated-error impact window ends after approximately 6 hours and 55 minutes.
• June 10, 17:30 UTC: Google updates its status page, confirming resolution for affected users. (Full service restoration was confirmed after 14 hours and 49 minutes).
• June 12, 11:14 UTC: Google publishes a preliminary incident report with early root-cause analysis.
• June 16, 17:41 UTC: Google publishes the final Incident Report outlining prevention measures.

Time to Detect (TTD): Not publicly disclosed. Google stated that internal real-time monitoring detected high processing error rates. 

Time to Resolve (TTR): ~ 7 hours impact window. Google also reported full service restoration after a total duration of 14 hours and 49 minutes. 

Who was affected?

The outage impacted both consumer users and enterprise Google Workspace customers. Users across the Gemini web app, iOS, and Android experienced persistent prompt failures, with a 50% error rate at peak impact. Affected users primarily received "Something went wrong" messages when attempting to send prompts. Gemini in Chrome suffered persistent timeouts within the Side Panel and integrated conversational features.

‍

Enterprise Workspace users lost the ability to attach files from Google Drive in the Gemini web application, with the "Add from Drive" functionality appearing disabled and not responding after users interacted with the prompt. 

How did Google respond?

Google detected the disruption via internal monitoring alerts. Engineering teams immediately targeted the overloaded database shards to reduce pressure and restore cache effectiveness.

‍

The initial response involved applying RateACLs to throttle traffic on the impacted shards, buying time for internal caches to stabilize. Engineers then redistributed common index values to evenly disperse traffic across a wider range of shards.

‍

To reduce database pressure during the incident, responders increased the cache TTL from 1 minute to 20 minutes. They also shortened RPC deadlines for database queries, stopping slow requests from accumulating and exacerbating the backend backlog. Google's long-term prevention strategies include restructuring the database index to prevent hotspotting, implementing adaptive cache policies, improving monitoring for uneven shard distribution, and adding smart task resizing and query coalescing.

How did Google communicate?

Google utilized the official Google Workspace Status Dashboard to communicate with users. The company posted regular updates confirming the investigation, mitigation efforts, and eventual resolution. Following the outage, Google issued a preliminary incident report on June 12, followed by a comprehensive final Incident Report on June 16 detailing the technical causes, customer impact, and remediation steps.

Key learnings for other teams

• Design indexes for even distribution: Account for repetitive values and empty identifiers to prevent traffic concentration on a small number of shards.
• Tune cache TTLs carefully: A one-minute TTL may improve freshness, but it can also increase database refreshes. During backend stress, adaptive cache policies can help prevent load amplification. 
• Monitor distribution, not just averages: Standard health signals often hide uneven shard utilization. Actively monitor for hot partitions and abnormal traffic concentration.
• Implement load protection early: Apply rate limits, query coalescing, and capacity-aware task sizing before backend services reach their breaking point.
• Optimize RPC deadlines during latency: Shorten RPC deadlines to prevent slow requests from piling up and prolonging recovery. 

Quick summary

On June 10, 2026, Google Gemini experienced elevated errors for approximately 6 hours and 55 minutes after a traffic spike pushed an already heavily utilized metadata database beyond its capacity threshold. An index design issue concentrated data on a few database shards, while a one-minute cache TTL contributed to a more than 10x surge in database calls. This caused severe read contention, database failure rates of up to 60%, and a 50% prompt error rate for users. 

How ilert can help

Incidents like Gemini’s demonstrate how quickly localized backend bottlenecks can cripple customer-facing AI services. ilert equips engineering and infrastructure teams to cut through the resulting alert noise and coordinate rapid responses.

‍

• Alerting on backend service issues: Route prompt error rates, database latency, cache hit rate drops, and shard-level load signals directly to ilert. Notify the right responders instantly to acknowledge and escalate issues.
• Reducing alert noise: Leverage ilert AI to group similar timeouts, database failures, and client errors from a single source.This reduces noise and helps responders focus on related symptoms. 
• Escalating to the right specialists: Build escalation policies and on-call schedules to notify database, SRE, and platform engineers when critical hotspot or cache health alerts trigger.
• Communicating service degradation: Utilize status pages and AI-generated updates to keep users informed during elevated error periods while engineers focus on mitigation.
• Turning incident data into action: Use AI-assisted postmortem generation to transform incident context, responder activity, and chat history into structured, actionable postmortem reports.

‍