How a large ISP fights DDoS attacks with a custom solution

DDoS attacks are a growing problem. In July, Arbor Networks released global DDoS attack data derived from its ATLAS threat monitoring infrastructure that shows a surge in volumetric attacks in the first half of 2014 with over 100 attacks larger than 100GB/sec reported.

Sadly, it’s not just the strength we should be worried about. In fact, BT found that DDoS attacks are becoming more effective, causing major disruption and sometimes bringing down organizations for entire working days.

We often hear about attacks against websites, most of which are mitigated by one of the many DDoS mitigation services available on the market. What I always wondered was how the big guys tackle these attacks. What weapons can an ISP bring to the battleground?

Technology made to fit

Sakura Internet, one of Japan’s largest Internet Service Providers, developed a technology that ingests massive IP traffic flow data streams and performs in-memory analytics to identify and stop DDoS attacks on its network as they happen, while simultaneously enabling legitimate traffic to continue.

They built a DDoS attack mitigation solution on the high-speed NewSQL database VoltDB to have the capability to analyze 48,000 IP packets per second, allowing them to see in real time which sites are under attack, and perform source-and-destination-based filtering, allowing clean incoming packets to move through, while blocking the bad.

In April 2014, the very first month of the solution’s production with VoltDB, Sakura detected and mitigated 60 DDoS attacks while also successfully restoring legitimate traffic to the majority of targeted sites – as the attacks were happening. Sakura was able to restore service in 49 of the 60 attempts it made, and in some cases, in as little as twenty seconds.

The man with the plan

The architect behind this solution is Tamihiro Yuzawa, a Network Engineer in charge of datacenter networking, inter-DC and inter-AS routing operations at Sakura Internet.

Yuzawa’s team is in charge of Sakura’s core network and his primary concern are large scale, bandwidth-hogging incoming DDoS attacks with junk packets that would not only disrupt legitimate communication to and from the victim host, but also cause collateral damage to other customers sharing their uplink with the attack target.

How damaging can DDoS be to an ISP? “If we fail to promptly mitigate an attack of the magnitude of 10Gbps or beyond, the consequence could be ugly route flaps in our network, which would seriously degrade our services,” says Yuzawa.

In most cases, it takes less than 10 seconds for the attack traffic to grow over gigabits per second. Before they developed this solution, it was only after their network monitoring system would dispatch alerts for reachability problems that they would look into some other information to find out there had been a DDoS attack already coming in.

Yuzawa remembers that, back then, sampled traffic data was already available, with a couple of collectors up and running. But traditional RDBMSs would easily saturate and choke on a huge influx of sampled data, and were thus unable to execute queries for real-time analysis.

“We had been quite aware that it would take a truly fast data processing backend that would import and compute a massive amount of data without delay so as to detect an attack right away. That’s why it jumped out at me when the release of VoltDB hit the news,” Yuzawa said.

Below is a screenshot that showcases a part of the app’s UI (click to see a bigger version). To capture this image, Yuzawa used archived traffic sample data to replay a recent DDoS attack. Only the destination IP address (in red) has been masked with the test server’s. While the attack continues at around 1.6~1.7Gbps, the blue line indicates the first step (RTBH) completed at 15:38:15, and the green line indicates the second step (cleaning) was activated at 15:38:23. And as of 15:45:27, 730 source IP addresses are being filtered.

There’s a lot of talk about how Big Data can help mitigate DDoS attacks, but how does it work exactly? Yuzawa explains that every router deployed in their backbone network is capable of sampling packets as they arrive in the router’s ingress interfaces, and encapsulating the sampled packets into a UDP message in a standardized format, which are then exported to the designated collector. The collector, acting as a database client, feeds the database with sampled traffic data as it arrives from the routers.

There are several other database client processes running all the time, one of which requests various types of aggregate queries per each destination IP address, so that it can identify a flow of malicious incoming traffic on the spot. Another client continues to profile each attack, including the origin IP addresses, also on the spot.

With the constantly updated information about each attack, the mitigation system protects legitimate traffic to the target host while blackholing attack packets.

When custom equals better

Yuzawa told me that over the years Sakura evaluated several commercial DDoS mitigation appliances, a couple of which showed impressive capabilities. However, in order to be able to execute truly effective countermeasures quickly, in the order of seconds, they needed to incorporate something very specific to their network design, configurations, and operations, and essentially create their own anti-DDoS solution.

You’ve built a custom solution that solves your information security problem? Drop me an e-mail.