Firebase: Config cleanup crashes iOS apps worldwide

Company and product

Firebase is Google's app development platform, which includes analytics and remote configuration for mobile apps. Google Analytics for Firebase (GA4F) ships as an SDK that developers embed directly in their apps.

The SDK uses remote configuration flags delivered from Google's servers, including flags Google uses as kill switches to revert bad releases.

What happened

Engineers removed a stale, legacy configuration flag from the server-side GA4F configuration. The change was two lines, and existing tests passed. But the iOS SDK still referenced that flag.

‍

The failure unfolded in four steps:

1. The flag was removed server-side. The cleanup produced a configuration payload that no longer contained an entry the SDK expected.
2. The payload rolled out globally. The flag mechanism is itself the kill switch Google uses to revert bad SDK behavior, and that mechanism is released globally.
3. The SDK did not validate its input. When the SDK parsed the payload, it did not check that a flag's name was non-nil. The missing flag triggered a fatal error, and the app crashed on launch.
4. Each device crashed once, then recovered. Active users who fetched the bad payload hit a one-time crash, after which the SDK fell back to the previous config. The long tail was in reporting, not recovery: crash reports were queued locally and only uploaded the next time a user successfully opened the app.

The core issue was a contract mismatch: the server treated the flag as removable, while the client treated it as required. Existing tests passed, and nothing in the pipeline caught the leftover reference.

Timeline

All times are Pacific Daylight Time (PDT), as given in Google's postmortem.

• September 28, 2026

• 17:38 PDT: A stale, legacy configuration flag is removed from the GA4F server configuration.
• 17:41 PDT: The malformed payload begins rolling out to production servers. Apps fetching it start crashing on launch.
• 17:59 PDT: Developer crash alerts spike and the first external bug reports appear on GitHub. Google's on-call engineers are notified.
• 19:16 PDT: Responders trace the crashes to the flag cleanup and start a rollback.
• 19:52 PDT: The rollback completes on all production servers. The bad payload is no longer served.
• Following hours: Crash reports stay elevated because devices queue reports and upload them the next time the app opens.

• October 2, 2026: Google publishes its postmortem on the Firebase blog.

Time to Detect (TTD): about 18 minutes (17:41 to 17:59). Google says it was alerted within 20 minutes through a variety of channels; its timeline shows developer crash alerts spiking and the first external bugs reported via GitHub.

‍

Time to Resolve (TTR): 2 hours 11 minutes from rollout to full rollback (17:41 to 19:52). Diagnosis took most of that time: about 1 hour 17 minutes passed between detection and identifying the culprit change.

Who was affected?

Google describes the impact as a "large number of applications on iOS and other Apple platforms" but has not published counts.

• App users on Apple platforms: Active users of apps embedding the GA4F SDK hit a one-time crash on launch during the window.
• App developers and their on-call teams: Crash rates spiked across unrelated apps at once.
• Crash and stability metrics: Delayed crash uploads kept crash reports elevated for hours after the fix.

Google says it did not observe any devices entering a permanent crash loop.

How did Google respond?

Once engineers linked the crashes to the flag cleanup at 19:16 PDT, they rolled the server configuration back to its last known-safe state. The rollback reached all production servers by 19:52 PDT and stopped the crashes without requiring an app update.

‍

Google committed to four remediation tracks:

1. SDK validation: A GA4F patch, planned within a week, strictly validates server payloads. Corrupt or incomplete flags are ignored, and the SDK falls back to cached defaults. Google will audit its other iOS and Android SDKs for similar parsing weaknesses.
2. Pipeline checks: Automated validation in configuration pipelines and pre-submit tests to block changes that leave orphaned identifiers or broken dependencies.
3. Safer rollouts: Investment in the configuration deployment process to improve monitoring and rollout phasing.
4. Status reporting: SDK-level outage signals on the status dashboards, and a less manual update process.

How did Google communicate?

Google's postmortem is open about where communication fell short.

• Status dashboards stayed green throughout. Both the Firebase and Google Ads dashboards rely primarily on server-side health metrics, so client-side SDK crashes never registered.
• No dedicated status entry existed. The Firebase dashboard has no separate GA4F component and links to the Ads dashboard instead. Updating both was manual and took several hours.
• GitHub became the de facto status page. With dashboards unreliable, Google posted updates in the public GitHub issue where developers first reported the crashes.
• Console banner and follow-ups. Google added a banner to the Firebase console and updated the dashboards the next morning and evening.

The public record only partly matches this. The Firebase status history lists no incident for September 28–29 and refers Google Analytics incidents to the Ads status dashboard. There, a Google Analytics incident covers 17:41–19:52 PDT on September 28 (00:41–02:52 UTC on September 29), matching the postmortem's window. Its first update, "We're investigating reports of an issue with Google Analytics," was posted at 17:57 PDT on September 29 (00:57 UTC on September 30), about 22 hours after the fix. That fits Google's evening update, but no morning update is visible. None of its updates mention iOS, the SDK or app crashes.

The postmortem itself, published four days later with timestamps, root cause and remediation, is a strong example of transparent follow-up.

Key learnings for other teams

• Treat config as code, and as a deployment. A two-line flag removal had global blast radius. Configuration changes need the same review, staging and rollback discipline as binaries.
• Clients must validate what servers send. An SDK running inside someone else's app should never crash on a missing or malformed field. Ignore bad input, fall back to cached defaults, and log it.
• Check the contract between server and client before removing anything. Before deleting a flag, confirm no shipped client version still reads it. Shipped mobile SDKs live on devices for months or years.
• Your kill switch needs its own safety net. The mechanism meant to revert bad releases was itself the thing that shipped globally. Emergency controls need phased rollout and validation too.
• Roll out progressively, even for "safe" changes. A canary to a small share of clients would likely have surfaced the crash quickly, at a fraction of the impact.
• Server health is not user health. The dashboards stayed green while apps crashed. Monitor client-side signals such as crash rates and SDK error telemetry, not only server metrics.
• Your status page must reflect reality. A green status page during a visible outage erodes trust. Make it easy to update, and wire client-side signals into it.

Quick summary

On September 28, 2026, Google removed a stale flag from the Google Analytics for Firebase server configuration. The iOS SDK still referenced that flag and did not validate its input, so apps on Apple platforms crashed on launch after fetching the new payload. The bad configuration was served globally for 2 hours and 11 minutes before a rollback, while status dashboards stayed green because they rely primarily on server-side health metrics. Google has committed to stricter SDK validation, pipeline checks, investment in config rollout phasing and better status reporting. The lesson for every team: treat configuration as code, validate on the client, and monitor what users actually experience.

How ilert can help

• Correlate alerts with recent changes: Link alerts to deployments and config changes so responders see "what changed" first, cutting the gap between detection and diagnosis.
• Escalate reliably: On-call schedules and escalation policies make sure the right engineer is reached in minutes, by push, SMS or voice call.
• Keep a status page that matches reality. ilert status pages can be updated in seconds or automatically from incidents, so customers are not left with a green page during a visible outage.
• Bring in dependency outages fast: When a third-party SDK or provider breaks your app, ilert helps you declare an incident, notify stakeholders and communicate, even when the root cause sits upstream.
• Speed up investigation with built-in AI SRE: ilert's AI SRE capabilities help responders pull together alerts, recent changes and context during an incident, and draft the postmortem afterwards.