Facebook down, worst outage in 4 years. Many suspected hackers

[Mark]

Facebook goes down! Outage effects millions


On September 23, 2010 - Facebook suffered their worst ever outage in 4 years.

The well known social networking site Facebook revealed that it had suffered the worst outage in four years, lasting two and a half hours. An estimated 500 million people use Facebook of which 135 million people were unable to log on to the site and instead were greeted with the following message: "Network Error (dns_server_failure)".

Facebook state that it was a key flaw in the handling of an error condition that caused the site so many problems, an automated system that was set up to fix the problem ended up making it worse.

Apparently, what kicked this of was an outage to a one of Facebook's major fibre lines, in Washington (WA). Facebook released the following statement: "We are experiencing an issue with a third party networking provider that is causing problems for some people trying to connect to Facebook.

In one of their later statements, they declare:

"Today we made a change to the persistent copy of a configuration value that was interpreted as invalid. This meant that every single client saw the invalid value and attempted to fix it. Because the fix involves making a query to a cluster of databases, that cluster was quickly overwhelmed by hundreds of thousands of queries a second.

To make matters worse, every time a client got an error attempting to query one of the databases it interpreted it as an invalid value, and deleted the corresponding cache key. This meant that even after the original problem had been fixed, the stream of queries continued. As long as the databases failed to service some of the requests, they were causing even more requests to themselves. We had entered a feedback loop that didn't allow the databases to recover.

"The only way to stop the feedback cycle was quite painful, we had to stop all traffic to this database cluster, which meant turning off the site. Once the databases had recovered and the root cause had been fixed, we slowly allowed more people back onto the site."

Millions of people posted comments about not being able to connect on blogs and other sites like Twitter.

[T3hRogue]

Millions of people posted comments about not being able to connect on blogs and other sites like Twitter.

Irony right there

[Drumm]

Just goes to show how reliant some people are on Social Media.

[Special Chris]

I saw this on The Escapist and people blamed it on Anon..o.o