Online Trends

#FacebookDown – what happened?

Yesterday wasn’t a good day for Facebook. The website, alongside with its subsidiaries WhatsApp and Instagram went offline without warning last night – and they remained down for several hours.

Needless to say, it quickly became a Big Problem. #FacebookDown was a trending hashtag all through the night.

What exactly happened to Facebook?

Once Facebook went down, network researchers went to work trying to determine the cause of the issue. The first and likely culprit was Facebook’s DNS servers.

DNS is the glue that holds the modern internet together. Thanks to the magic of DNS, you get to type in “sentione.com” instead of 213.186.34.10 every single time. When you break DNS, addresses stop resolving and you can’t connect to your service.

The problem turned out to be slightly deeper than just DNS, however. An issue in BGP configuration brought Facebook down. If you think of DNS as a paper map, then BGP is your phone’s GPS app: picking the most efficient route from your computer to any network destination.

Facebook, as any tech company of its size, relies on their own BGP implementation. This is what triggered the outage – or rather, a faulty configuration file pushed to production (the “live” environment”).

Within moments, Facebook’s network stopped responding.

The problem also spread to Instagram and WhatsApp. While these companies are Facebook subsidiaries, the outage shouldn’t have affected them – they are hosted on Amazon’s AWS, outside of Facebook data centres.

And yet, these services also went down. This is why the Facebook outage escalated into a full blown #InternetShutDown, as christened by Twitter users.

If you’re interested in more details, CloudFlare posted a detailed writeup of the outage and its causes.

How did the Facebook outage affect the internet?

To us, the Facebook outage might seem like something blown out of proportion. After all, we have access to so many other social networking sites, on which we can (and did!) poke fun at Facebook’s network engineers.

"#FacebookDown" was the biggest trending hashtag of last night - with almost a million tweets to its name.

But for a lot of the world, Facebook is the entire internet. In developing countries, Facebook’s infrastructure and investments help provide cheap internet access to dozens of millions of people. The outage affected them the most – suddenly, people in those countries found themselves cut off from the wider internet.

This demonstrates the biggest flaw in the modern internet. By relying on massive companies to provide access to wide swaths of the internet, we’re going against the basic design principle of the internet. The original design called for a decentralized network: that way, not even a nuclear strike would bring the network down. Avoiding single points of failure ensures reliability.

Facebook being the sole internet access provider for millions of people introduced a single point of failure. This is the same story as the Fastly outage from earlier this year.

On a front closer to home, Mark Zuckerberg himself found his wealth affected by the outage: within a couple of hours, his net worth dropped by $7 billion. Even regular Facebook employees were hit: the system they use to control the electronic locks across the Facebook campus also went down!

Was SentiOne affected by the Facebook outage?

Thankfully, the outage didn’t affect us in a major way. While it is one of our primary data sources, it is by no means the only one. 

We kept on trucking, gathering social listening data from other sources. When Facebook came back online, we resumed receiving data from the site without any issues. In fact, when looking at our graphs, we can pinpoint to the hour the moment Facebook started coming back online. Take a look for yourself:

Around midnight (CEST) we started receiving data from Facebook again: the issue was fixed.

How did social media react to the Facebook outage?

As it turns out, social media users weren’t too displeased with Facebook going down. On the contrary! The outage was met with humour and a healthy dose of schadenfreude at the misfortune of Facebook’s network engineers.

13.70% of all mentions of the outage were positive in tone. Only 6.93% eexpressed a negative sentiment.

Twitter in particular was busy: that’s where most of the discussion about the outage was held, with posts about the topic reaching over 137 million impressions.

Internet users took to Twitter en masse to discuss the Facebook outage.

And where there’s schadenfreude, there’s memes. Here’s some of our favourites:

Even Tommy Wiseau, arguably the greatest filmmaker of all time, got on the action: