The real cost of alarm fatigue

One of the toughest challenges for an IT security team is managing and sifting through the deluge of security alerts that are created on a daily basis. Teams can waste considerable time chasing down false positives with the resultant burden on manpower and IT resources adding costs. However, the stakes are high; failing to detect an active infection can have far more serious financial consequences.

Alarms and alerts are designed to signal problems that need attention. However, when alarms are constant, and a high percentage of these are false positives, there is a real danger that we can become de-sensitised to their importance. Given this rising tide of continuous alerts, it’s perhaps of little surprise that teams are facing alarm fatigue – a situation in which teams can be overwhelmed by information and constant notifications which means that real threats become the needle in the haystack that is ignored or missed.

With organisations managing thousands of alerts each day, it’s more important than ever that organisations can cut through the noise and have confidence in the accuracy and effectiveness of solutions to detect even the most sophisticated threats.

A tsunami of alerts

The failure of antivirus to detect the latest malware and threats, has spawned an entire industry of alerting technologies, from Intrusion Detection products and black boxes deployed in the network, highlighting vague anomalies, to sandboxes and threat intelligence solutions spewing Indicators of Compromise (IoCs) incessantly into our operations.

This unending tsunami of alerts related to infections and compromises, and incidents and breaches, has then created another industry of alert consolidation, correlation tools, coalescence technology, big data analytics, attribution biased triage – and the list goes on. With limited time and manpower to deal with these threats, it’s inevitable that this information overload will place additional burdens on teams. As a result, it´s not just alarm fatigue teams are suffering from but, budget and resource fatigue also.

However what choice do they have? If the endpoint is not instrumented precisely, there is no choice but to cobble together as many diverse technologies as possible, in the hope that something will trigger an alert accurate enough to prevent escalating harm to our internal systems. This happens in the name of best practise, continuous monitoring, vigilance and resilience and this is the reality of IoC- led alerts in security operations.

Risks across the chain

There are risks on both sides of the equation: we know we can have too many alerts, and we can’t investigate them all, but there is also a risk in uninvestigated alerts. There is risk in the amount of time it takes to diagnose whether an IoC based alert is real and present in the environment, and if it is, determining if it was successful, or blocked by another security function somewhere in the environment.

If we consider a chain of events, like the kill chain, where an attack moves from one layer to the next, exploit becomes infection, successful infection becomes compromise, unchecked compromise becomes incident, and an incident with sensitive data exfiltration becomes a breach, there is the potential for a false positive and false negative or misdiagnosis at every stage in that chain.

IoC alerts are, at the end of the day, simply just indicators and the more sources of indicators there are, the greater the potential for alert fatigue at every stage in the kill chain. Alerts based on indicators are responsible for the majority of lost time in security escalations. What we alert on, needs to change; alerts need to be based on actualities in the environment, not possibilities.

Of course, the most precise place to measure actualities in the environment is the target of the attack itself, the endpoint. This is now a real alternative to the existing IoC alert avalanche that props up our ineffective antivirus. The dynamic behaviour of the endpoint is the security actuality that should form the basis of a real alert. Modelling the dynamic behaviour of the endpoint, solves the problem of protection bias, which is the fundamental flaw in security controls by which attackers are able to infiltrate our environments.

The answer to alert fatigue, and unsatisfactory risk management, lies in the move away from indicative alerts, and the problems of volume, inaccuracy and cost that it brings. We should, instead move towards the goal of alerts based on actual target system behaviours. A security escalation process based on “something is happening”, instead of “something might be happening”.