A comparative view of cloud-based DDoS protection services

Six months ago we experienced a 30Gb/sec and 60M PPS attack that was targeting over 1000 IPs on our network. Although we eventually stopped the attack with the aid of our upstream providers, a number of our customers asked us why we didn’t have a DDoS protection service in place. We decided on NTT’s service due to their scale and network capacity. However, this solution was meant only to protect our network in times of need, and not to protect individual customers on a 24/7 basis. One customer revealed that above all else, DDoS attacks are what keep him up at night.

When it comes to specialized DDoS mitigation service providers, we only had some basic information from a handful of our customers. We passed this along when asked, but we hadn’t done any formal comparisons. It was time to change that.

First, we needed to research the marketplace which we narrowed down to a few top contenders. We chose two of the largest players out there today: CloudFlare and Incapsula. We offer a range of Internet infrastructure services in seven locations globally: Vancouver, Seattle, LA, Toronto, NYC, Miami, and London UK. However, our main office is based in Vancouver BC, so we included DOSarrest, a local Vancouver-based company, to round out three different options to compare. Full disclosure, I worked at PEER1 Hosting for many years, and DOSarrest’s CTO headed my department. Since they’re local to us, we decided to also see how they compare to the two big dogs.

Notes
It’s clear that all three participants are in constant change/upgrade mode. We trialed each of them for one to three months and within this period they all had enhanced their service offering and/or dashboard. Incapsula added new graphs and upgraded some components, CloudFlare unveiled a completely new customer portal and DOSarrest upgraded so many different components their dashboard looked completely different then when we started our demo 30 days earlier. It shows this is an evolving field, as the attacks change so do the cloud-based DDoS protection companies.

This article contains our observations, comments and recommendations. I’m sure other organizations would see, experience and rate everything differently. Given the complexity of these services, it would have been a major undertaking to test out every option available, and that was not our intention. Our goal was to get a taste of their services so we would be comfortable recommending them to our customers.

The participants
Cloudflare: Launched in late 2010 as a CDN with cloud based DDoS protection services, has evolved into a number of other services and has customers numbering a million+. They do offer a free subscription that does not include DDoS protection.

Incapsula: Started in 2009 with the backing of Imperva, a security hardware manufacturer who has since acquired a majority interest in Incapsula.

DOSarrest: Started in 2007 as one of a handful of companies at the time specializing in cloud based DDoS protection services.

Under each category we rated the participants from 1-10, 10 being best. With each participant, we wanted to choose their tier or level of service that included a DDoS protection service, CDN for performance, and a WAF – all at a comparable protection level. Easier said than done.

CloudFlare
We chose their Business Option, which allows you to run one website on their service. Trying to get straight answers to simple questions proved to be a little more complicated than we anticipated. “How much clean bandwidth can you run?”, Cloudflare’s response “It’s unlimited”. When pressed further, they told us they measure bandwidth by the number of simultaneous connections.

“What is the maximum number of connections you can run?” The answer we received was: “We can’t tell you for security reasons”. We gave them a 9.0 as they were the lowest cost provider at $200/month if you only have one URL to protect. Their next tier of service starts at approximately $3,000-$5,000/month.

Incapsula
We chose their Enterprise account, which offers 25Mb/sec of clean bandwidth and 1 Gb/sec of DDoS protection for $500/month. Should you be unfortunate enough to be the victim of an attack over 1 Gb/sec, the attack traffic is sent back to your origin and you have to sign up for a one year contract on a tier of service that can accommodate the attack.

Given that your monthly cost could go up substantially if you had a 8 Gb/sec attack or larger, we gave them a 7.

DOSarrest
This participant has only one tier of service, 10 Mb/sec of clean traffic. They only charge for traffic between your website and your visitors, not traffic between them and your server. They guarantee 200Gb/sec of attack traffic protection. There’s one caveat: they only protect website traffic. In other words HTTP and HTTPS TCP ports 80 and 443. You can’t run your mail server or DNS through their system. We gave them a 7.5 as there are no possible surprise costs that could hit you even if you experience a large attack.

1. Provisioning/setup
Cloudflare: They’re very different than the other two participants. They use CNAMES, so you first have to add a TXT entry into your DNS records, to prove to them that you control the DNS for your domain. Once that’s done, they will give you a CNME to point your domain to, after that you are good to go. The CNAME is broadcast out of all of their nodes, and distributed around the globe. I really can’t see how some of our customers could easily navigate this process with speed and ease, especially under the stress of a DDoS attack. We gave them a 7.0.

Incapsula: They assign a unique IP to point the A record to in your DNS for the domain you want to protect. They then anycast this IP on three of their nodes. Although they have 20+ global mitigation nodes, we only seemed to be using three of them, given our location in Vancouver they selectively broadcast our unique IP out of Seattle, San Jose and Los Angeles as far as we could tell. Pretty straight forward and easy, we gave them an 8.5.

DOSarrest: They gave us a virtual IP which you point your domain to and they in turn anycasted this IP out all of their available scrubbing nodes, there are only four: London, NYC, LA and Singapore. Very easy and smooth setup, we gave them an 8.0.

2. The dashboard
Cloudflare: During our testing they had one main dashboard with very minimal analytics and graphs. They did however have a second portal available that was in beta at the time, which was much better, so I’ll discuss it.

Their new dashboard comes stock with some metrics. They focus on one metric, which is requests cached and non cached, the graph has a modern design, loads quickly and has historical statistics for the last 30 days. There are numerous widgets on the dashboard where you can view and make changes.

There are so many different widgets each with selectable items, options and sub-options, it can be somewhat difficult to remember how to get back to where you were. Overall the dashboard is fast, easy to view, and has many tool-tips as well as some supplemental screens you can click, giving you more information on a particular function/option. We gave them an 8.0 on their dashboard, took a point off for being a little busy.

Incapsula: The dashboard is very easy to navigate. Clicking on the various categories brings you to a views with more reporting, analytics or configuration screens. In general the dashboard is well designed and responsive, while some of the stock reporting seemed crammed in a bit. We gave them an 8.5.

DOSarrest: Very different feel, no widgets! If you want to configure something you click configure. Joking aside, we thought it was very easy to view. Pretty much all of the analytics/reporting is also on the main dashboard view. You can go with the stock graphs, etc. or select from 13 different reporting visuals to have on your dashboard. The more you choose means you just have to scroll down, nothing is crammed in like with the others. We rated them an 8.5 as we thought it was the easiest to view and understand.

3. Reporting and analytics
Cloudflare: Easy to read and view, the most basic of the participants. Analytical traffic reporting consists of total requests cached and un-cached, top threat IPs by country, top 5 countries of clean requests, and top 5 search engine traffic sources. Also up for viewing are total threats stopped, types of threats, and percentage of SSL traffic served. On the business plan you can only get stats for the last 6 hours, so seeing a real-time impact on any of these graphs/displays may not be apparent. You have a choice of the last 6, 12 or 24 hours, last week or last month. We gave them an 8.0, we took points off for lack of visual real-time reporting. Some of the information provided on limited screen real estate, such as search engine activity, was of little use.

Some sample Cloudflare screenshots:

Incapsula: There was a good selection of traffic analytics, which included: visits, hits, bandwidth, requests, and a breakdown from which country they come from. There are performance metrics related to cached bandwidth and requests, and from which Incapsula node they are being served from. We couldn’t get a single TCP traceroute to end up in their Dallas node, although the display says 80% of our traffic was being sent out from Dallas.

They had a single view threat page that we also liked where you could see all the threats and which type; you could even drill down for more details on each threat. There was an events page which had the same info but in a log style format in real-time where you could select on the source of the event. This was useful as we could focus in on WAF violations alone. It could get very busy though, and it seemed there was a little too much info on some views. Where Incapsula really shined in our opinion was that they had 30 days worth of historical reporting, not just a screen shot of last month’s data, with fast access. We gave them a 9.5 because some of the views only had a limited amount of items, the top 10 IPs were there, but some of the information was missing.

A few sample screenshots from Incapsula:

DOSarrest: These guys have the best design for reporting. You can toggle any metric or variable on or off on a graph to see the remaining metrics better, you can also get any of these stats based on any one of their nodes. The best thing about it? It’s all on one page – you select the graphs you’re interested in and it’s all displayed on one view, just keep scrolling to see them all. We took off points for historical (30 days+) reporting, which is by request, we gave them a 8.5.

A few sample displays from DOSarrest:

4. Configuration/customization
This is the most complicated item to evaluate as it can be as simple or as involved as you like, so I’m only going to give my general observations on the whole procedure.

Cloudflare: Good IP whitelisting and blacklisting page – you can block or present a captcha for black listed IPs. There are many different icons, pages and subsections with options, which makes it difficult to get back to where you were to undo a change, or even view it again. The WAF section alone has literally 3,000+ signatures/items you can toggle on or off! To get the most out of their system you will need to invest some serious time going through the various pages. We gave them an 8.0 because there are just too many things and widgets some of which are not related to DDoS protection or even security.

Incapsula: Configuration additions, changes and modifications are easy to use, almost instantaneous, and are described well. The security options are numerous and require some experimentation to understand. We only tried a few of the options available, but all seemed to function as advertised. There are many different views or screens to make changes and modifications, which can be confusing. Their WAF is much less complicated compared to Cloudflare’s, but it does require some time to master. We gave them an 8.5.

DOSarrest: Their configuration screens had an industrial look and feel to them, we didn’t understand it at first, they call their options “features”. Once we found out how it worked, it all clicked. First you pick a location to apply a feature, you can choose the entire website site or a particular URI to apply the feature. Features are divided into two categories Security or Performance. With the other participants most of their options are applied to the entire website being configured.

The other thing we liked was that we could view what was being applied from one screen, sounds simple but with Cloudflare we would have to go through 15-20 views/screens not including sub-options to see what if any options were turned on, not counting the WAF. If you have to manage multiple sites for multiple customers you don’t want to have to click 25 screens to see what’s on or what’s off. We gave them a 9.0 because of the simplicity and the fact they will actually do any configuration changes for you. They also said they can pretty much create a custom feature for you within 24 hours or less.

5. DDoS protection
We tried to simulate a small DDoS attack as we knew we did not have the firepower to overcome any of them but we did test it somewhat. We went for a layer 7 attack and used a combination of JSLOIC and a web stress tool. No surprise it had zero effect on the website whatsoever on any of the participants. We actually received an email from a real person at DOSarrest during testing telling us there was an attack, and our site was unaffected. Incapsula had it recorded in one of their online reports as well. Everyone gets 9.0.

6. Performance
On each participant we enabled the maximum amount of caching available. On Cloudflare we were unable to use their option “Railgun” (some sort of caching enhancement) as it requires a piece of code to be installed on your server. With Incapsula we used “aggressive” caching, which will override any cache control headers on your website and cache for a specified time interval. On DOSarrest we used a feature called “forced caching” which is similar to Incapsula’s “aggressive caching”. To perform the tests we used the Keynote systems standard 5 city test to measure performance, which measures load times and provides a detailed view of the time for every element on a webpage. We ran 10 tests in a row every day over a 10 day period. 10 tests X 5 cities X 10 days+ 500 samples from each. Some of the samples were way out so we just used a sample in each region. Cloudflare had some very wild fluctuations compared to DOSarrest and Incapsula.

We broke it down by region as our server of origin is on the west coast of Canada. All of the performance was pretty close. The results were so close we gave everyone an 8.5.

7. Monitoring

Cloudflare: There is no real-time performance monitoring of your website provided by Cloudflare. There is however an option to get basic monitoring through a third party (Pingdom). You have to register with them and pay extra for any serious monitoring. We gave them a 6.0.

Incapsula: Their monitoring consisted of testing availability of the website from three of their nodes to the three nodes we were running on. They have since beefed this up and now have a more comprehensive performance/availability monitoring system but our demo was already finished and we didn’t see it. We gave them a 7.5.

DOSarrest: They have the best system, completely separate from all of their nodes. It tracks response time, uptime, content changes, and SSL expiration. It calculates % uptime and other stats, and has up to one year of historical data. It’s fast and even has a smartphone app available. Notifications are sent by the 24/7 SOC and not through an automated system. However, because it’s a completely different system, you are redirected through the dashboard to another service website. Nonetheless, we gave them an 8.0.

8. Support

Cloudflare: Very good email support! We needed help many times to get things working. We sent 10 different emails/tickets to their NOC and never waited more than 10 minutes for a response. In fact, some were answered in less than 5 minutes ! We would’ve given them a 10, but since there’s no phone support on the business package, we gave them a 9.0.

Incapsula: We only used their tech support once and entered a high priority ticket which was because we couldn’t figure out how to turn off a captcha that we had enabled for testing purposes. It was an easy fix for them but it took 50 minutes to get a response back. We gave them a 7.5.

DOSarrest: We used their support email and ticket system 5 times. We always received an answer within 15 minutes. Everything is fully managed, and on one occasion they went into our configuration and made the change for us, then notified us. They do have phone support, but we never used it. We gave them an 8.0.

Overall impression

Cloudflare: Their support by email/ticket system was great! I have never seen such consistent fast replies, from any service period. Our overall impression was that there was more steak than sizzle on the system as a whole. There are so many screens, options, add-ons, etc. it was a little confusing and complicated.

DOSarrest: This was the big surprise for us. We didn’t expect too much, but found it easy and hassle free from start to finish. Their traffic analytics were the best, and because it’s fully managed you don’t even ever have to login. Their performance monitoring was best of the group, and the fixed cost was also a big plus.

Incapsula We liked Incapsula, our techs found their dashboard easy to work with, and their weekly report would go over well with some of our customers. The only drawback was when were told that if an attack exceeded 1Gb/sec they would reroute the traffic back to us until we re-signed a revised one year agreement at a higher tier of service to handle the attack. The support was not as speedy as we would have liked.

Conclusion and recommendations
All of these participants have vast experience in dealing with DDoS attacks and are dealing with an amazing amount of granular data, which enables them to analyze and stop even the most sophisticated attacks. As is always the case, it’s a matter of price/performance and service/responsiveness and how comfortable you are with leaving your prized possession in their capable hands.

We would recommend theses DDoS protection companies if the customer fits the requirements outlined here:

Recommend Incapsula:

• Customer has some technical skills
• 2-3K+ a month budget (base protection is only 1Gb/sec at $500.00/month)
• More than 5 -10 domains to protect
• Does not require blazing fast support response
• Prefers to deal with a larger established organization

Recommend DOSarrest:

• Customer has limited time or technical skills –it’s a fully managed service
• Has a need for fast(10-15 min) phone and email support
• Fixed budget with no surprises – there’s only 1 tier of service. $800.00/month
• Comfortable with a smaller organization
• Less than 10 domains to protect

Recommend Cloudflare:

• Customer has limited budget ($200.00/month per URL)
• High technical skills
• Likes a feature rich environment
• May have a need for add-on services
• Does not require immediate setup/protection less than 15 minutes
• Very high risk customer, may be a target of 100Gb+/sec attacks
• Has clean traffic in the 50Mb/Sec + range
• Does not require phone support
• Has only 1 or 2 main URLs to protect.

Overall scoring recap:

We hope you found this information useful and encourage you to contact any of the three participants should you find yourself or your customers in need of a cloud based DDoS protection service.