Google Cloud: Generators fail to start, taking a London zone offline

Company and product

Google Cloud is Google's public cloud platform. Its europe-west2 region is located in London and is split into three zones: europe-west2-a, europe-west2-b and europe-west2-c.

‍

Google's report separates two kinds of impact. Zonal resources, such as Compute Engine VMs, Persistent Disk and non-high-availability database instances, live in a single zone. Regional services span the zones in a region, and their control planes can depend on components in other zones.

What happened

The data center hosting europe-west2-b is designed with fully duplicated utility power (two feeds, A and B), a generator bank controlled by a dual-redundant programmable logic controller (PLC), and uninterruptible power supply (UPS) batteries to bridge the gap while generators start. A standby utility connection can also be switched in manually, campus-wide, during maintenance or emergencies. From October 14, one feed was locked out for repairs, and on October 19 (US/Pacific) the remaining layers failed in sequence.

‍

1. One feed was already out. After issues identified during a quarterly generator load test on October 14, circuit breaker feeder B was physically opened and key-interlocked for repairs. The site was running on feed A alone.
2. The remaining feed failed. At 22:53 US/Pacific on October 19 (06:53 BST on October 20), feed A was lost because of an external underground cable fault.
3. The generators never started. The PLCs that command generator start checked whether voltage was present on the feeders, but not whether the feeder breakers were closed. Their sensors were wired upstream of the locked-out feed B breaker, where the line was still live. Both the primary and the backup PLC read that as a healthy utility source, so no start command was sent. Google identifies this misconfiguration of the PLC logic as the root cause.
4. The batteries ran out. The UPS batteries provide about 5 minutes of runtime, meant to cover the time for the PLC to detect undervoltage, start and synchronize the generators, and transfer the load. Feed A dropped at 22:53, and by 23:00 the batteries were depleted and the zone lost power.

The failure then spread beyond the zone. Neighboring zones europe-west2-a and europe-west2-c, and regional services, were hit by demand surges and cross-zone control-plane dependencies. Google says some failovers were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.

The pattern is a latent common-mode fault exposed by degraded-mode operation. The PLC logic flaw only mattered once maintenance moved the site from its normal 2N utility setup to a single feed, and because both redundant PLCs shared the same logic and inputs, they failed the same way.

Timeline

All times are US/Pacific, as given in Google's report. London time (BST) was 8 hours ahead.

• October 14, 2026: Issues are identified during a quarterly generator load test. Following this, feeder B is opened and key-interlocked for repairs.
• October 19, 2026
  
  • 22:53: Utility feed A is lost due to an external cable fault. The PLCs do not start the generators.
  • 23:00: Impact begins as the UPS batteries are depleted and europe-west2-b loses power.
  • 23:04: Monitoring alerts Google engineers.
  • 23:09: Automated regional failover begins.
  • 23:54: Power from the standby utility source is restored.
• October 20, 2026
  
  • 00:20: Data hall power is restored.
  • 00:40: Clusters begin recovering.
  • 00:45: Network devices are back online.
  • 00:50: Storage recovery begins.
  • 01:35: Google posts its first status update.
  • 03:13: Storage is sufficiently restored.
  • 03:45: Controlled traffic re-admission to europe-west2-b begins.
  • 04:11: Google Cloud services and APIs are fully available, with some Compute Engine VMs still unavailable.
  • 04:29: Regional services are fully undrained.
  • 04:46: New VM creation resumes.
  • 05:08: Majority of workloads recovered, with over 99% of VMs healthy.
  • 07:03: Closure notification posted.
• October 22, 2026: Google publishes its preliminary incident report.
• October 27, 2026: Google publishes its incident report with root cause and preventive actions.

Time to Detect (TTD): 4 minutes (23:00 to 23:04). Google says engineers were alerted by monitoring. (The preliminary report gave 23:08 for the alert.)

‍

Time to Resolve (TTR): 6 hours 8 minutes (23:00 to 05:08). Google reports an initial phase of 5 hours 29 minutes (23:00 to 04:29) in which zone-scoped resources in europe-west2-b were unavailable, followed by a recovery tail for VM creation and remaining workloads.

‍

Power itself was back after about 1 hour 20 minutes (data halls at 00:20). Most of the outage was spent recovering storage, clusters and traffic after the unclean shutdown.

Who was affected?

Google's report lists 40 affected products across europe-west2 and says it found no impact outside the europe-west2 region.

• Zonal resources in europe-west2-b: Compute Engine VMs, Persistent Disk and Hyperdisk, GKE zonal clusters and nodes, non-HA Cloud SQL and AlloyDB instances, Cloud SQL read replicas in -b, and single-cluster Bigtable instances were unavailable.
• The rest of the region: Neighboring zones and regional services saw latency, HTTP 500 and 503 errors, admission queuing and transient storage write errors, driven by demand surges and cross-zone control-plane dependencies.

The impact reached beyond Google Cloud's own products: at least one banking app reported a temporary service disruption that it attributed to the Google Cloud outage, and later confirmed that services were restored.

How did Google respond?

After the PLC blocked a remote load transfer, the onsite facilities team transferred the facility load to the standby utility source, which was supplying power by 23:54 US/Pacific. Data hall power was back at 00:20. Recovery then took several more hours: storage devices had to recover from unclean shutdowns, responders deliberately held back restoring services that would have made that recovery worse, and traffic was re-admitted to the zone in a controlled way, starting at 03:45. After the outage, staff reconfigured and validated the PLCs.

‍

Google's report commits to preventive actions in five areas:

• Utility power: repair the external cable fault and return the site to its normal two-feed setup.
• Generators and PLCs: repeat generator load transfer tests, review the PLC code with vendors, validate PLC telemetry, and check PLC configurations at sites with similar designs worldwide.
• Procedures: update the emergency operating procedures and practice them twice per shift.
• Change management: freeze planned maintenance at similar sites and require stricter approvals for non-standard changes.
• Cloud services: speed up customer notifications for incidents that affect a whole location, and expand platform testing for infrastructure-level events.

How did Google communicate?

Detection was fast, but the first public update came much later.

• Internal alert in 4 minutes. Monitoring alerted engineers at 23:04 US/Pacific, 4 minutes after impact began.
• First status update after 2 hours 35 minutes. Google's first post on the status page went up at 01:35 US/Pacific (09:35 BST). It reported issues with multiple services in europe-west2. By then, data hall power had already been restored.
• The affected zone was named after 5 hours 12 minutes. Updates at 01:35, 02:24 and 04:00 US/Pacific described elevated errors and intermittent network failures across europe-west2. The first update naming a power event in europe-west2-b came at 04:12, almost 4 hours after data hall power was back. Until then, customers deciding whether to move workloads out of zone b did not know it was the affected zone.
• Practical guidance in updates. Status updates advised customers to migrate workloads to other zones or regions where feasible and asked impacted customers to open support cases.
• Closure and follow-up reports. Google posted a closure notification at 07:03 US/Pacific, a preliminary incident report on October 22, and a full incident report on October 27.

One item on Google's action list concerns customer notifications. Under communications, Google plans to speed up notifications by building an automated mechanism that turns alerts into customer communication for incidents affecting a whole location.

From 07:00 to 09:35 London time, the start of the UK working day, there was no public status update.

Key learnings for other teams

• Redundancy is only as strong as its current state. The site was designed with two feeds and generators, but a maintenance lockout left it on one feed. Track when planned work reduces redundancy, and treat that window as elevated risk.
• Make sure sensors measure what the load sees. The PLCs checked for voltage but not breaker position, and read it upstream of an open breaker, so they saw power that the data halls did not have. A control signal is only useful if it confirms the thing you are protecting.
• Redundant controllers that share a flawed input fail together. Google ran a primary and a backup PLC, but both used the same logic and the same upstream sensing, so neither started the generators. Redundancy only helps against faults that the copies do not share.
• Test the backup path in degraded configurations. A backup chain that works with both feeds in place may behave differently when one is locked out. Include maintenance states in failover testing.
• Restoring power is not restoring service. Power was back in about 1 hour 20 minutes, but storage, clusters and traffic took hours more, and responders held back services that would have slowed storage recovery. Plan and rehearse recovery from an unclean shutdown, including what not to restore first, not just the switchover.
• Zonal failover needs headroom. When one zone failed, demand surges and cross-zone dependencies hit the other two, and some failovers failed on capacity. Multi-zone designs need spare capacity and as few shared control-plane dependencies as possible.
• Know your provider's blast radius. If a single zone can take down your login, payments or APIs, that is a dependency to map, monitor and plan around, especially in regulated industries.
• Close the gap between detection and communication. Google detected the outage in 4 minutes, posted publicly after 2 hours 35 minutes, and named the affected zone after 5 hours 12 minutes. Customers need both speed and scope: which zone is down is what lets them decide whether to fail over.

Quick summary

On the morning of October 20, 2026 (London time), the Google Cloud data center hosting europe-west2-b in London was running on a single utility feed because the other was locked out for repairs. When that feed failed due to an external cable fault, the generators did not start: the control logic checked for voltage but not breaker position, and its sensors sat upstream of the locked-out breaker, where the line was still live. The UPS batteries ran out and the zone lost power, while failover traffic caused transient latency and errors across the region. At least one banking app attributed a temporary service disruption to the outage. Google mitigated the primary impact after 6 hours 8 minutes and has committed to PLC fixes and validation at similar sites, updated emergency procedures, a maintenance freeze with stricter approvals, and faster automated customer communication. The lesson for every team: test redundancy in the state you actually run in, and plan for what happens after the power comes back.

How ilert can help

• Detect it yourself: When your provider's status page is silent, your own alerts are your detection. On-call schedules and escalation policies make sure the right engineer is reached in minutes, by push, SMS or voice call.
• Keep a status page that matches reality: ilert status pages can be updated in seconds or automatically from incidents, so your customers are not left waiting hours for the first update, even when your provider is.
• Speed up investigation with built-in AI SRE: ilert's AI SRE capabilities help responders pull together alerts, recent changes and context during an incident, and draft the postmortem afterwards.