Facebook’s in-house drills were crucial for smooth response to watering hole attack

Facebook was the first company to admit publicly to have been affected by the recent watering hole attack that started with a compromised forum site popular with mobile developers.

Their security team has not only been the first to discover the presence of the malware in its networks and to identify it, but has also discovered the source of the infection and the C&C server that the malware was sending information to.

After having sinkholed the server and analyzed the incoming traffic, they discovered that other companies have been affected, too, and they proceeded to notify them. In the end, it turned out that Apple, Twitter and Microsoft were among these companies.

But what allowed Facebook to react so well to the breach?

At the CanSecWest Conference held this week in Vancouver, Ryan McGeehan, security manager for incident response at Facebook, and Chad Greene, manager of the Facebook CERT, shared details about how Facebook prepares its security teams for incidents such as that one.

The first, dubbed “Vampire”, was a simulation of a full breach in which actors that pretended to be affiliated with the Koobface gang had apparently owned a number of laptops across the company (including a few of those belonging to members of the security teams), installed backdoors, had root access to the server, were actively disrupting the security teams’ log searches, and ultimately sent an email containing screenshots proving the “ownage” and blackmailing the company.

As they prepared to completely rebuild the domain in order to assure that the attackers are booted out, the news came that it was all just a drill.

While the “attack” was unfolding, the incident response team kept an eye on all people involved in the defense, evaluating their reactions and their interactions. As McGeehan pointed out, there is no other way to prepare effectively for incidents such as these apart from mounting this type of exercises – first-hand experience is crucial.

The second “attack”, dubbed “Loopback” and executed at the end of 2012, was more similar to the real one that hit the company at the beginning of this year, Threatpost reports.

With the help of an in-house developer who was instructed to let malware in by clicking on a link in a spear-phishing email that exploited a zero-day vulnerability, the “attackers” established control of the system and used it to push out altered PHP code that created a backdoor in Facebook’s production code.

This change would have affected all Facebook users, were it not for the fact that the backdoor was not designed to run.