Effectiveness of web application firewalls

The best defense against web application vulnerabilities and threats is to professionally train Web Application Firewalls (WAFs) and to incorporate Dynamic Application Security Testing (DAST) software products that create automatic filters, a recent analyst study found. The study also found that Intrusion Prevention Systems (IPS) tuned with DAST filters also creates an effective WAF.

Security industry expert Larry Suto conducted a benchmark study with eight WAFs and IPSs, and evaluated their relative effectiveness in detecting, reporting and thwarting web attacks.

Each of the eight systems was evaluated using two separate tests. The first tested how effective the IPS or WAF performed against external attacks if the configuration was tuned in one day or less by an experienced security professional. The second test looked at how each IPS or WAF performed when trained by DAST solution (NTOSpider) generated filters.

The study tested each solution against the same set of websites and web application prototypes to ensure the experiments were instantiated against well-known and well-understood vulnerabilities.

A number of conclusions were derived from the tests:

• When tuned only with “out of the box’ network specific rules, IPS solutions were not very effective at defending web application vulnerabilities. However, when trained by DAST generated filters, IPS solutions improved by an average of 60 percent, bringing up their performance at-par or better than the trained/configured WAFs; with their overall blocking effectiveness averaging 82 percent.
• Baseline tuned WAFs are fairly effective at detecting and defending web attacks. The most effective solution found 88 percent of the vulnerabilities known in the test application; the average effectiveness across all solutions was 79 percent. An average of 19 percent more vulnerabilities were blocked when DAST generated filters were applied to WAF solutions.
• The study found that a highly trained expert required an average of 3.5 hours in order to tune the WAF or to an acceptable blocking level, which is significantly more time than what the typical organizations spends.

“WAFs can be a very valuable part of an organization when properly tuned and effectively trained with DAST filters, saving time and dramatically improving their effectives.” says Suto “I also found that IPS solutions, though not designed out-of-box for web application security, can be trained to be very useful as part of a broader security strategy or WAF alternative”

The study examined a cross-section of modern WAFs and IPSs, both proprietary and open source. The WAFs and IPS devices included in the testing are: WAFs Barracuda 360, Citrix NetScaler, DenyAll rWeb, F5 ASM, Imperva SecureSphere, ModSecurity, Sourcefire Next-Generation IPS and an un-named IPS solution.

The study used one DAST product, NTOSpider, which is capable of creating filters for more of these defensive products. NT OBJECTive’s NTODefend product was used for filter generation.