Facebook Goes Dark Worldwide for Six Hours After BGP Misconfiguration Removes Its Routes
What happened
A misconfigured BGP update during routine maintenance withdrew all Facebook IP prefixes from the global routing table, making Facebook, Instagram, and WhatsApp unreachable worldwide for six hours. DNS resolution also failed because Facebook's authoritative DNS servers were behind the same withdrawn routes.[1]
What went wrong
The BGP update that triggered the outage was part of a planned maintenance operation, but the audit tool that should have caught the error had a bug. Because all Facebook infrastructure depended on the same BGP routes including internal tools and VPN, engineers could not remotely diagnose or fix the issue.[1]
Lesson learned
Out-of-band management access must be independent of the infrastructure it manages. DNS and network infrastructure should have external fallback routes. Change-validation tooling must itself be tested — bugs in safety systems nullify their purpose.
Sources
- [1] Cloudflare / Facebook Facebook Goes Dark Worldwide for Six Hours After BGP Misconfiguration Removes Its Routes