Postmortem Library

GitHub: Stale routing records send database writes to retired servers, twice in one day

On October 7, 2026, a routine machine replacement in GitHub's database orchestration briefly sent writes for several core databases to retired database servers. In the first outage, writes recovered within about 10 minutes, after which GitHub paused automated maintenance, but an automated task did not honor that pause and triggered another election, which caused a second outage later the same day. This postmortem covers how outdated records, an automatic leader election and validation that let incorrect settings through combined into the first outage, why a second outage followed, and what teams can learn about routing changes and maintenance safety controls.

Company and product

GitHub is a platform for hosting code and collaborating on software. GitHub publishes monthly availability reports that summarize the incidents that degraded its services in the previous month.

‍

GitHub's report describes a set of machines that manage database write routing: they decide which database servers receive writes for each core database. One machine in the set is elected leader and publishes the routing settings that production traffic follows. These machines are replaced as part of routine maintenance, some of which runs automatically.

What happened

GitHub describes the outages as exposing an unsafe interaction between its database orchestration and production routing. The first outage unfolded in four steps, and the second outage repeated steps 2 to 4.

‍

  1. A replacement machine started with outdated records. Routine maintenance was replacing the machines that manage database write routing. One new machine started with records that still listed retired database servers as valid destinations.
  2. It was elected leader. The new machine was automatically elected leader of the set, which gave it control of the routing settings.
  3. It published wrong routing settings. As leader, it published settings that sent writes for several core databases to the retired servers.
  4. Validation let the settings through. The incorrect settings passed GitHub's existing validation checks and took effect immediately, so writes to those databases began to fail.

The first outage ended when an established machine with correct records became leader and published correct settings. GitHub traced the cause to the new machine and paused automated maintenance to prevent a repeat.

‍

The second outage. The new machine had not yet been removed from the set. Later that day, an automated maintenance task did not honor the pause and triggered another leader election. The new machine became leader again and the same failure repeated until an established machine took over again. GitHub then removed the new machine.

‍

The pattern is stale state promoted by automation. Outdated records on a new machine became a problem only once an automatic election gave it authority, and validation did not catch the bad output. The second outage adds a second pattern: a safety control (the maintenance pause) that the automation it was meant to stop did not honor.

Timeline

All times are Coordinated Universal Time (UTC), as given by GitHub. Below, an outage is the window when writes failed, and an incident is the record GitHub lists in its report.

  • October 7, 2026
    • 15:14: First incident. GitHub's report lists it from 15:14, lasting 1 hour 11 minutes. The first write outage lasted about 10 minutes; the report does not give clock times for the write outage itself. Automated monitoring alerted on failing writes and GitHub declared an incident.
    • After the first outage: An established machine with correct records becomes leader and writes recover. GitHub traces the cause to the new machine and pauses automated maintenance.
    • 16:52 to 17:01: Second outage, per GitHub's incident updates. An automated maintenance task does not honor the pause, triggers another leader election, and the new machine becomes leader again. Writes recover when an established machine becomes leader again.
    • 17:17: Second incident. GitHub's report lists it from 17:17, lasting 47 minutes. GitHub removes the new machine before resolving this incident; the report gives no time for the removal.
  • November 12, 2026: GitHub publishes its October availability report with a combined summary of both outages.

The report's incident durations (1 hour 11 minutes and 47 minutes) are longer than the write outages it describes (about 10 and 9 minutes). GitHub does not explain the difference.

‍

Time to Detect (TTD): not stated by GitHub. For the first outage, automated monitoring alerted on failing writes. For the second outage, GitHub does not say how it was detected.

Time to Resolve (TTR): writes recovered after about 10 minutes in the first outage and about 9 minutes in the second (16:52 to 17:01), roughly 19 minutes of write impact in total. GitHub's report lists the two incidents as lasting 1 hour 11 minutes and 47 minutes.

Who was affected?

The outages hit write operations across GitHub. GitHub says viewing content was largely unaffected.

  • Core write paths: Customers may have seen errors when pushing code, signing in, updating pull requests or issues, or making changes through the API.
  • Dependent services: Webhooks, GitHub Apps and OAuth (Open Authorization), Actions, code scanning, Codespaces and Copilot were also affected.
  • Scale: Web and API request failures peaked at 8.9% in the first outage, affecting more than 12,000 organizations, and at 9.3% in the second, affecting more than 11,000 organizations. Some organizations were affected by both.

GitHub says no data that had already been saved was lost, and actions that failed during the outages succeeded when retried after recovery.

How did GitHub respond?

In both outages, writes recovered when an established machine with correct records became leader and published correct routing settings. After the first, GitHub traced the cause to the new machine and paused automated maintenance. After the second, it removed the new machine from the set. A later update on the second outage said mitigations had been applied and GitHub did not expect another recurrence.

‍

GitHub's report says it has made changes in three areas:

  • Server retirement: improved how database servers are retired, so outdated records are not left behind.
  • Routing machines and maintenance: tightened controls on adding new machines that manage database routing, and on making sure automated maintenance honors a pause.
  • Validation: strengthened validation of database routing settings, so incorrect settings are caught before they take effect.

How did GitHub communicate?

  • An incident for each outage. GitHub opened a separate incident record for each. For the second outage, it posted updates in its community incident thread.
  • The second outage was named as a recurrence. GitHub's update for the second outage gave the impact window (16:52 to 17:01), stated that all systems had recovered, identified it as a recurrence of the first outage, and noted that the root cause was being investigated and durable mitigation was in progress.
  • Timing of the second incident record. The report lists the second incident from 17:17, 16 minutes after writes had recovered at 17:01.
  • Combined write-up about five weeks later. GitHub's October availability report, published November 12, explains both outages together with root cause, impact and follow-up actions. The report covers eight October incidents that degraded GitHub services, including the two October 7 incidents, and says the month's incidents, including the write outages, showed that capacity growth must be paired with safer operations. It also states that reliability was still not where it needed to be, and that its principle remains availability, then capacity, then features.

Key learnings for other teams

  • Retire state, not just servers. The retired databases were gone, but records pointing to them survived on a new machine. Decommissioning should remove every reference, and new machines should be checked against current state before they join.
  • Leader election is a deployment. Whoever wins an election gets to change production routing instantly. Treat a newly joined node as untrusted until it has proven its state is current, before it can lead.
  • Validate meaning, not just format. The wrong settings passed existing checks. Routing validation should confirm that destinations are live, healthy and expected, not only that the settings are well formed.
  • A pause must stop everything. The maintenance pause was meant to prevent a repeat, but an automated task did not honor it. Kill switches and freezes need to be enforced centrally and tested, not left to each tool to respect.
  • Remove the faulty machine before resuming. The faulty machine was still in the set when the next election ran. Until the trigger is removed, a recovery is a workaround, not a fix.
  • Small percentages can mean big blast radius. Peaks of 8.9% and 9.3% sound modest, but they hit write paths for more than 12,000 organizations in the first outage and more than 11,000 in the second. Measure impact by who could not work, not only by error rate.
  • Say when it is a recurrence. GitHub's update explicitly linked the second outage to the first. That helps customers who already planned around the first outage understand what changed.

Quick summary

On October 7, 2026, an unsafe interaction between GitHub's database orchestration and production routing caused two write outages. Routine maintenance on the machines that manage GitHub's database write routing introduced a new machine with outdated records that still listed retired database servers. When it was automatically elected leader, it published routing settings that sent writes for several core databases to those servers, and the settings passed existing validation. In the first outage, writes failed for about 10 minutes until an established machine took over. GitHub paused automated maintenance, but an automated task did not honor the pause, triggered another election, and the still-present machine caused a second outage of about 9 minutes. Web and API request failures peaked at 8.9% in the first outage, affecting more than 12,000 organizations, and at 9.3% in the second, affecting more than 11,000. GitHub says it has since improved how it retires database servers, tightened controls on new routing machines and on honoring maintenance pauses, and strengthened validation of routing settings. The lesson for every team: decommission state as carefully as servers, validate what settings mean, and make sure a pause actually stops automation.

How ilert can help

  • Turn failing-write alerts into the right responder: GitHub's first outage was caught by automated monitoring on failing writes. On-call schedules and escalation policies make sure the right engineer is reached in minutes, by push, SMS or voice call.
  • Keep a status page that matches reality: ilert status pages can be updated in seconds or automatically from incidents, so customers know what is affected, including when an incident is a recurrence.
  • Speed up investigation with built-in AI SRE: ilert's AI SRE capabilities help responders pull together alerts, recent changes and context during an incident, and draft the postmortem afterwards.
Find more Postmortems:
Ready to elevate your incident management?