GitHub: Unpinned dependency causes 36,800 Copilot code review failures

Company and product

GitHub is a software development and collaboration platform used by developers and organizations to host repositories, manage source code, review changes, automate workflows, and coordinate software projects.

GitHub Copilot is the company's AI-powered development assistant, offering features like code completion, chat, coding agents, and automated code review.

Copilot code review analyzes proposed changes in pull requests and provides feedback on potential bugs, quality issues, and possible improvements. Because this feature operates as a processing workflow, it depends on several components—job scheduling, runtime environments, software dependencies, and model services—working together. A failure in one processing dependency can prevent the entire review from completing.

What happened

The incident began when the workflow responsible for processing GitHub Copilot code reviews automatically consumed a newly released dependency. This dependency release introduced an incompatibility with the workflow’s runtime environment, preventing review processing from operating correctly.

Because the workflow was configured to automatically use the latest available version rather than a specifically tested and pinned version, the external release acted as an unvalidated automatic production change. Existing validation processes failed to block it. Once the new version was selected, Copilot code review jobs immediately began failing.

The issue was heavily amplified by the system's failure behavior. Jobs that could not successfully start or complete did not terminate immediately. Instead, they remained active until hitting their configured timeout limits.

This created a two-step recovery challenge: GitHub had to first stop new reviews from consuming the bad dependency, and then wait for thousands of previously started jobs to time out and drain from the processing queues. As a result, newly requested reviews began recovering while backend cleanup was still actively ongoing.

Timeline

• 17:30 UTC: Customer impact began. GitHub Copilot code review requests submitted through GitHub.com started experiencing elevated failures.
• During the incident: Affected pull requests displayed the message “Copilot ran into an error.” The average failure rate reached 81.6%, peaking at 93.9%.
• During the investigation: GitHub engineers identified a newly released dependency used by the code review processing workflow as the source. They determined it was incompatible with the workflow’s runtime environment.
• During mitigation: Engineers removed the problematic dependency version and redeployed the affected processing service.
• 18:44 UTC: Newly requested Copilot code reviews began recovering.
• 18:55 UTC: The code review failure rate returned to its normal baseline, ending the customer-facing impact.
• 19:59 UTC: Remaining review jobs that had been waiting to time out finished draining from the processing system.

Time to Detect (TTD): Not publicly disclosed.

Time to Resolve (TTR): 1 hour and 25 minutes for customer-facing impact, from 17:30 to 18:55 UTC. Complete backend queue drainage took 2 hours and 29 minutes.

Who was affected?

• Users requesting GitHub Copilot code reviews through GitHub.com.
• Approximately 36,800 code review requests failed.
• Users relying on automatic Copilot reviews experienced missing or delayed reviews.
• GitHub Enterprise Cloud with data residency was not affected.
• The outage was strictly limited to Copilot code review; core GitHub and other Copilot services remained fully operational.

How did GitHub respond?

After tracing the elevated failures to the incompatible dependency version, GitHub engineers removed the problematic release and redeployed the affected processing service. This successfully isolated new reviews from the incompatible code, restoring customer-facing availability by 18:55 UTC. Previously started jobs were left to run until their timeouts expired, clearing completely by 19:59 UTC.

To prevent recurrence, GitHub announced several systemic improvements:

• Pinning the dependency to a known compatible version instead of automatically consuming the latest release.
• Adding compatibility checks for all future dependency versions.
• Improving fast-failure behavior when the review processor cannot start.
• Introducing shorter timeout controls for review workflows to prevent capacity drain.
• Improving monitoring specifically for code review completion failures.

How did GitHub communicate?

GitHub documented the incident in its official June 2026 availability report. The communication was transparent, providing specific data on incident duration, failure rates, the volume of failed requests, and the technical cause.

Notably, GitHub clearly distinguished between customer-facing recovery at 18:55 UTC and complete backend recovery at 19:59 UTC. This distinction provided customers with an accurate, highly technical understanding of the incident's lifecycle, rather than relying on a single, misleading “resolved” timestamp.

Key learnings for other teams

• Pin production dependencies: Automatically consuming the latest dependency version turns an external release into an uncontrolled production deployment. Always use explicitly tested and pinned versions.
• Treat dependency updates as deployments: Pass all dependency changes through testing, staged rollouts, and monitoring before they reach production.
• Validate runtime compatibility: A dependency may install successfully but fail in the actual production runtime. Tests must confirm the complete workflow starts and finishes successfully.
• Fail fast when startup fails: A processing job that cannot initialize correctly should terminate immediately. Leaving failed jobs active until a long timeout wastes resources and slows down recovery.
• Set shorter, stage-specific timeouts: One long timeout for an entire workflow delays failure detection. Different processing stages require realistic, specific timeout limits.
• Monitor completed outcomes: Infrastructure health metrics alone won't reveal if customers are actually receiving results. Monitor the ratio of requested, started, completed, and failed jobs.
• Separate service recovery from queue recovery: Always track and report customer-facing restoration independently from complete backlog drainage.

Quick summary

On June 4, 2026, GitHub Copilot code review experienced a major failure spike that affected approximately 36,800 review requests and reached a peak failure rate of 93.9%. The incident was caused by an unpinned software dependency that automatically updated to a version incompatible with the workflow’s runtime environment. Failed jobs did not terminate immediately and instead remained active until timing out, consuming processing capacity and delaying complete queue recovery. GitHub removed the incompatible dependency version and redeployed the affected service, restoring customer-facing operations in 1 hour and 25 minutes, while the remaining backend work took 2 hours and 29 minutes to drain completely. The incident demonstrated that every unpinned dependency can become an external production deployment pipeline that a team does not fully control.

How ilert can help

When a dependency issue causes automated jobs to silently fail, teams must detect the drop in customer outcomes quickly without getting overwhelmed by alert noise.

• Alerting on review failures: Send critical metrics like completion rates, queue depth, and processing latency directly to ilert's reliable and actionable alerting to catch outcome drops even when infrastructure looks healthy.
• Escalating to responsible teams: Use ilert's on-call management and escalation policies to instantly notify specific platform, runtime, and dependency-management teams when thresholds are breached.
• Reducing duplicate notifications: ilert’s deduplication and alert grouping prevent thousands of failed jobs from triggering thousands of separate pages, keeping responders focused.
• Coordinating the rollback: Utilize dedicated incident channels to help developers and incident commanders coordinate testing, redeployment, and monitoring in real time.
• Communicating degraded functionality: Leverage ilert Status Pages to instantly communicate partial degradation, such as Copilot code reviews failing, while assuring users that the broader platform remains active.