Most security products fail to perform

Nearly 80 percent of security products fail to perform as intended when first tested and generally require two or more cycles of testing before achieving certification, according to a new ICSA Labs report. The “ICSA Labs Product Assurance Report” – co-authored by the Verizon Business Data Breach Investigations Report research team – details lessons gleaned from testing thousands of security products over 20 years.

The report found the number one reason why a product fails during initial testing is that it doesn’t adequately perform as intended. Across seven product categories core product functionality accounted for 78 percent of initial test failures. For example, an anti-virus product failing to prevent infection and for firewalls or an IPS product not filtering malicious traffic.

The failure of a product to completely and accurately log data was the second most common reason. Incomplete or inaccurate logging of who did what and when accounted for 58 percent of initial failures.

The report findings suggest that logging is often considered a nuisance and undervalued. According to the report, logging is a particular challenge for firewalls. Almost every network firewall (97 percent) or Web application firewall (80 percent) tested has experienced at least one logging problem.

Rounding out the top three is the startling finding that 44 percent of security products had inherent security problems. Security testing issues range from vulnerabilities that compromise the confidentiality or integrity of the system to random behavior that affects product availability. Even though it can be a demanding process, certification with a trusted, established third party is critical to verifying product quality, states the report. Product categories studied were: anti-virus, network firewall, Web application firewall, network IPS, IPSec VPN, SSL VPNs and custom testing.

George Japak, managing director, ICSA Labs said: “The question I ask vendors is this: Who would you rather have find an issue in your product — ICSA Labs in a safe testing environment or a criminal in the real world?”

In addition to product functionality, logging and inherent security problems, other issues identified in the study include poor product documentation and patching. Poor product documentation is unhelpful and dangerously misleading. The report indicates vendors should place more importance on proper documentation.

Additionally, patching remains an issue. Approximately 20 percent of products struggle to accept updates correctly. For products like anti-virus, the ability to accept patches effectively is as important as the product’s core functionality of preventing infection.

Recommendations for enterprises

While some of the report findings may be startling, there are steps that companies should take before purchasing and using security products. Key recommendations include:

• Use certified products. While certification can never eliminate risk, it substantially reduces risk by ensuring that products meet objective, publicly vetted criteria.
• Demand quality. The market typically prefers features over quality. If end users demand quality, vendors will supply it.
• Place certification at the top of the list of desired product characteristics. When shopping for a product, start with a list of certified products and then compare based on features, price and other important characteristics.
• Be suspicious of performance claims and numbers. Vet them. Question them. Be an educated, cautious buyer.
• Choose more established products over new. New products have more problems; often times the kinks have not been worked out yet. Choose more established products when possible. If a new product is required, make sure the product is certified.
• Choose simplicity over complexity.
• When using a certified product, keep up with whether the certification is current. If the vendor loses or does not maintain certification, determine why. The answer may affect the organization’s security posture.
• Push vendors to maintain certification after end-of-life. If a product is still in use, quality assurance is important.
• Prefer vendors that are certified. Working with consortia and industry organizations is a positive factor for product quality. Report findings show that vendors who participate in such groups enjoy a higher level of performance in testing. Technology held to an industry standard improves in both features and reliability.
• Expect the unexpected.

The ICSA Labs report is available here.