CircleCI: Network routing issues disrupt macOS jobs

Company and product

CircleCI is a continuous integration and continuous delivery platform that helps software teams automate build, test, and deployment workflows.

For macOS jobs, CircleCI relies on a third-party Mac infrastructure provider. These environments support teams building and testing software for Apple platforms and require reliable access to external services such as GitHub. During this incident, the failure did not come from CircleCI’s core platform or GitHub directly, but from the network path between CircleCI’s Mac infrastructure provider and GitHub.

What happened

Starting at 18:00 UTC on July 13, 2026, jobs running on CircleCI’s macOS fleet began to stall during GitHub fetch steps. Typically, these fetch processes take one to two minutes; during the incident, they extended to 20–30 minutes and frequently resulted in job timeouts.

The root cause was traced to a capacity and routing problem on specific network paths between CircleCI’s Mac infrastructure provider and GitHub. CircleCI’s macOS jobs are hosted by a third-party Mac infrastructure provider, which connects to the broader internet through multiple upstream network paths. When some of those paths experienced reduced capacity or routing problems, macOS jobs fetching code or dependencies from GitHub slowed down or failed to progress.

The incident was intermittent. The first occurrence appeared to recover after traffic was rerouted, but the issue returned later the same day and recurred again on July 14. CircleCI later confirmed that its existing automated monitoring did not catch the incident, which exposed a gap in end-to-end monitoring for GitHub fetch performance on the macOS fleet.

The most important reliability lesson is that the failure did not sit neatly inside one platform. It lived in the seam between providers. That made it harder to detect, reproduce, and close confidently.

Timeline

• June 24, 20:18 UTC to June 25, 03:09 UTC: CircleCI experienced a similar macOS fleet incident affecting the ability of macOS jobs to reach GitHub. That incident was also caused by a network routing problem involving the Mac infrastructure provider.
• July 13, 18:00 UTC: Some customers running macOS jobs began experiencing intermittent failures and delays when fetching code and dependencies from GitHub.
• July 13, 18:55 UTC: CircleCI opened an investigation and alerted customers through its status page.
• July 13, 19:04 UTC: CircleCI’s Mac infrastructure provider identified increased latency on one network path and rerouted traffic around it.
• July 13, 19:40 UTC: CircleCI marked the initial occurrence as resolved.
• July 13, 20:44 UTC: Customers reported that the issue had returned.
• July 13, 21:52 UTC: CircleCI reopened the incident and moved the status page back to investigating.
• July 13, 23:35 UTC: CircleCI published workaround guidance recommending that customers increase no_output_timeout to 15–20 minutes and moved the incident to monitoring.
• July 14, 13:59 UTC: CircleCI confirmed that customers were still experiencing intermittent failures and moved the status page back to investigating.
• July 14, 15:35 UTC: CircleCI updated customers that its infrastructure provider was testing network configuration changes to isolate the source of the issue.
• July 14, 16:31 UTC: CircleCI reported that reduced inbound bandwidth on a specific route was negatively affecting repository fetch times.
• July 14, 16:36 UTC: The Mac infrastructure provider identified affected network paths and shifted traffic away from them.
• July 14, 16:58 UTC: Fetch times recovered, and CircleCI moved the incident to monitoring.
• July 14, 20:18 UTC: Some customers again experienced slow fetch times, and CircleCI moved the status page back to investigating.
• July 14, 22:07 UTC: The provider identified two additional degraded network paths and shifted traffic away from them.
• July 14, 23:18 UTC: Fetch times stabilized, and CircleCI marked the incident resolved.

Time to Detect (TTD): Not publicly disclosed. Customer impact began at 18:00 UTC, and CircleCI opened the public investigation at 18:55 UTC. CircleCI later stated that its existing automated monitoring did not catch this incident.

Time to Resolve (TTR): Approximately 29 hours and 18 minutes from the start of customer impact to final resolution, with intermittent impact during that window.

Who was affected?

The incident affected some customers running jobs on CircleCI’s macOS fleet, especially workflows that needed to fetch code, repositories, or dependencies from GitHub.

Affected jobs hung or failed to progress during GitHub fetch steps. Fetches that normally took one to two minutes stretched to 20–30 minutes for some customers and often timed out.

How did CircleCI respond?

CircleCI opened an investigation, alerted customers through its status page, and worked directly with its third-party Mac infrastructure provider to identify degraded network paths and shift traffic away from them.

After the issue returned, CircleCI reopened the incident and gathered diagnostic data with its provider. The team also published workaround guidance recommending that affected customers increase no_output_timeout to 15–20 minutes, giving slow GitHub fetches more time to complete while the underlying network issue was still being mitigated.

On July 14, CircleCI reproduced the failure using its own testing tools, confirming that the issue was a general git-fetch problem rather than something tied to a specific build tool, container image, or software update. The provider later identified additional degraded network paths and shifted traffic away from them before the incident was finally marked resolved.

How did CircleCI communicate?

CircleCI used its public status page to keep customers informed throughout the incident. Updates explained when the issue was under investigation, when traffic had been rerouted, when the incident moved to monitoring, and when the problem recurred.

CircleCI also published workaround guidance recommending that affected customers increase no_output_timeout to 15–20 minutes, giving slow GitHub fetches more time to complete while the underlying network issue was still being mitigated.

After the incident, CircleCI published a post-incident report explaining the recurring routing issue, the prior June incident, the monitoring gap, and the planned prevention work.

Key learnings for other teams

• Monitor the inter-vendor path: Standard platform monitoring can miss failures that happen between external providers. If a workflow depends on GitHub fetches from a third-party Mac fleet, that exact path needs end-to-end monitoring.
• Treat recurrence as signal, not coincidence: A similar macOS-to-GitHub routing issue had happened weeks earlier. Recurrence is strong evidence that the failure pattern needs a structural fix, not only incident-by-incident mitigation.
• Avoid premature incident closure: The issue returned after an initial resolution. Intermittent network incidents need a confirmed clean-monitoring window before being marked fully resolved.
• Build diagnostics before the next incident: CircleCI built diagnostic tooling during the incident to reproduce the git-fetch failure. Turning that tooling into a standing capability can reduce investigation time next time.
• Reduce single-provider concentration risk: CircleCI’s plan to move from a single Mac infrastructure provider to multiple providers directly addresses the risk exposed by this incident.

Quick summary

For nearly 30 hours, some CircleCI customers running macOS jobs experienced severe delays and timeouts when fetching code and dependencies from GitHub. The root cause was degraded capacity and routing on network paths between CircleCI’s third-party Mac infrastructure provider and GitHub.

The same class of macOS-to-GitHub routing issue had occurred several weeks earlier, making this more than a one-off outage. The incident exposed a recurring inter-vendor path dependency that CircleCI’s existing automated monitoring did not catch.

CircleCI mitigated the issue by working with its provider to reroute traffic away from degraded paths. Its prevention plan includes end-to-end GitHub-fetch monitoring, faster provider-side degraded-path detection, permanent diagnostic tooling, longer confirmation windows before closing network incidents, and a move toward multiple Mac infrastructure providers.

Reliability principle: If your customer workflow depends on traffic crossing provider boundaries, monitor the seam, not just the systems.

How ilert can help

Incidents like CircleCI’s show how difficult it can be to coordinate response when the failure sits between providers, not inside one clearly failing system. Once monitoring or synthetic checks detect degradation, ilert helps teams route alerts, escalate quickly, and communicate clearly during intermittent recovery.

• Alert routing after detection: Send macOS job failures, GitHub fetch latency, VCS timeout rates, and synthetic git-fetch check failures from monitoring tools to ilert, so the right responders are notified when CI workflows begin to degrade.
• Escalating to the right teams: Use on-call schedules and escalation policies to notify CI platform, infrastructure, networking, and vendor-facing responders when fetch delays or job timeouts cross critical thresholds.
• Communicating degraded CI performance: Use status pages to keep customers informed when builds are delayed, when a workaround such as increasing timeout settings is available, and when teams are monitoring for recurrence.