#FacebookDown – what happened?
Yesterday wasn’t a good day for Facebook. The website, alongside with its subsidiaries WhatsApp and Instagram went offline without warning last night – and they remained down for several hours.
Needless to say, it quickly became a Big Problem. #FacebookDown was a trending hashtag all through the night.
What exactly happened to Facebook?
Once Facebook went down, network researchers went to work trying to determine the cause of the issue. The first and likely culprit was Facebook’s DNS servers.
So, @facebook‘s DNS is broken this morning…
TL;DR: Google anycast DNS returns SERVFAIL for Facebook queries; querying https://t.co/0BDgaIHmlr directly times out. pic.twitter.com/3GHJ3mW0P0
— Jim Salter (@jrssnet) October 4, 2021
DNS is the glue that holds the modern internet together. Thanks to the magic of DNS, you get to type in “sentione.com” instead of 213.186.34.10 every single time. When you break DNS, addresses stop resolving and you can’t connect to your service.
The problem turned out to be slightly deeper than just DNS, however. An issue in BGP configuration brought Facebook down. If you think of DNS as a paper map, then BGP is your phone’s GPS app: picking the most efficient route from your computer to any network destination.
Facebook, as any tech company of its size, relies on their own BGP implementation. This is what triggered the outage – or rather, a faulty configuration file pushed to production (the “live” environment”).
Within moments, Facebook’s network stopped responding.
Live replay of Facebook disappearing off the internet.#facebookdownhttps://t.co/4SeWPvBWNF pic.twitter.com/U5gbgcPtRd
— Kim Karlsson (@ParanoidVoxel) October 4, 2021
The problem also spread to Instagram and WhatsApp. While these companies are Facebook subsidiaries, the outage shouldn’t have affected them – they are hosted on Amazon’s AWS, outside of Facebook data centres.
And yet, these services also went down. This is why the Facebook outage escalated into a full blown #InternetShutDown, as christened by Twitter users.
If you’re interested in more details, CloudFlare posted a detailed writeup of the outage and its causes.
How did the Facebook outage affect the internet?
To us, the Facebook outage might seem like something blown out of proportion. After all, we have access to so many other social networking sites, on which we can (and did!) poke fun at Facebook’s network engineers.
But for a lot of the world, Facebook is the entire internet. In developing countries, Facebook’s infrastructure and investments help provide cheap internet access to dozens of millions of people. The outage affected them the most – suddenly, people in those countries found themselves cut off from the wider internet.
This demonstrates the biggest flaw in the modern internet. By relying on massive companies to provide access to wide swaths of the internet, we’re going against the basic design principle of the internet. The original design called for a decentralized network: that way, not even a nuclear strike would bring the network down. Avoiding single points of failure ensures reliability.
Facebook being the sole internet access provider for millions of people introduced a single point of failure. This is the same story as the Fastly outage from earlier this year.
1990: The Internet is a magical distributed system that instantly routes around any instability, surviving lines being cut, servers crashing, and even nuclear war.
2021: Three companies own the internet and one of them broke.
— Daniel Feldman (@d_feldman) October 4, 2021
On a front closer to home, Mark Zuckerberg himself found his wealth affected by the outage: within a couple of hours, his net worth dropped by $7 billion. Even regular Facebook employees were hit: the system they use to control the electronic locks across the Facebook campus also went down!
Facebook employees can’t enter the headquarters because their badges don’t work, and those already inside can’t enter various rooms because access is linked through the IoT (Internet of Things) and so goes through the same DNS routes that no longer exist:#FacebookDown pic.twitter.com/8hAea9ZG4l
— Leah McElrath 🏳️🌈 (@leahmcelrath) October 4, 2021
Was SentiOne affected by the Facebook outage?
Thankfully, the outage didn’t affect us in a major way. While it is one of our primary data sources, it is by no means the only one.
We kept on trucking, gathering social listening data from other sources. When Facebook came back online, we resumed receiving data from the site without any issues. In fact, when looking at our graphs, we can pinpoint to the hour the moment Facebook started coming back online. Take a look for yourself:
How did social media react to the Facebook outage?
As it turns out, social media users weren’t too displeased with Facebook going down. On the contrary! The outage was met with humour and a healthy dose of schadenfreude at the misfortune of Facebook’s network engineers.
Twitter in particular was busy: that’s where most of the discussion about the outage was held, with posts about the topic reaching over 137 million impressions.
And where there’s schadenfreude, there’s memes. Here’s some of our favourites:
People making jokes about the Facebook services going down make me sick. Anyone who was using an Oculus headset at the time is currently trapped in VR, and if they die there then they die in real life.
— Gavin Young (@GavinDYoung) October 4, 2021
BREAKING: live view inside a Facebook datacenter pic.twitter.com/GylYGW0wHU
— xXx_ANT1F4_5C3N3_QU33N_xXx (@pupOnSecurity) October 4, 2021
Facebook: Goes down.
Anti-vaxxers: pic.twitter.com/nk6fqnkrCg
— Drac U. Law, Vampire Attorney (@PinkRangerLB) October 4, 2021
Even Tommy Wiseau, arguably the greatest filmmaker of all time, got on the action:
Facebook is down !
Instagram is down !
Welcome to my twitter ! pic.twitter.com/SaP8gBHL28— Tommy Wiseau (@TommyWiseau) October 4, 2021