The Outage That Wasn’t Human Error

On January 25, 2023, an engineer ran a command on a non-production router, triggering a global WAN outage impacting all Microsoft cloud services. The simplistic “human error” story masks a far more instructive reality: the incident resulted from a chain of governance gaps, flawed mental models, and unvalidated SOP changes over two months.

The command reset the IGP routing table globally, but only because the SOP was created and modified outside normal change review—no peer review, no emulation testing. The engineer trusted the SOP, believing unsafe commands would be blocked by AAA. The team treated non-production routers differently, skipping listening periods and safety checks.

The real failure lies in systemic deficiencies: inconsistent SOP governance for non-production assets, missing command audits during onboarding, and a culture that equates “following the SOP” with safety. Reactive fixes like adding training or punishing the engineer would have ignored these deeper issues, risking repeat incidents.

Engineering leaders should note that the tactical repairs (e.g., making BGP less sensitive to IGP churn) addressed the immediate failure mode, but the lasting fix required embedding this incident into onboarding training to reshape mental models. The talk underscores the need for blameless analysis that surfaces systemic factors, not just quick closure metrics.