The evolution of security analytics

As networks continue to evolve and security threats get more complex, security analytics plays an increasingly critical role in securing the enterprise. By combining software, algorithms and analytic processes, security analytics helps IT and security teams proactively (and reactively) detect threats before they result in data loss or other harmful outcomes.

Given that the average time to identify and contain a data breach in 2021 was 287 days, it’s more important than ever for organizations to include security analytics in their threat detection and response programs. But how has this technology changed over the last decade? In this article, I will explore the evolution and importance of security analytics.

This evolution has had two main trends.

First, security analytics is becoming more sophisticated. In the last 10 years the industry has transitioned from rule-based alerting to big data and machine learning analysis. Second, products have become more open and customizable.

As these technologies have advanced, so too have their specific use cases, with organizations using these for identity analytics (examining authentication, authorization and access for anomalies), fraud (finding anomalous transactions), and more. Today, security analytics plays a central role in Security Information and Event Management (SIEM) solutions and Network Detection and Response products (not to mention standalone security analytics software).

To better understand this evolution and the capabilities of current security analytics solutions, let’s dive into the three primary generations of security analytics advancement.

Generation One

Traditional security analytics focused on correlation and rules within a proprietary platform.

Users imported data into a closed database, the data was normalized and run through a correlation engine, and then the system produced alerts based on rules. Products typically included alert enrichment, which provided more useful context along with an alert, such as linking it to a specific user, host, or IP address.

However, this era often suffered from “alert fatigue” where the analytic solution produced more alerts than the security team could investigate, including high numbers of false positives. Sorting which alerts were important and which ones weren’t involved a great deal of manual work. Furthermore, these solutions were often entirely proprietary, with little to no options for customization. This prevented the security team from tweaking rules to cut down on the number of bad alerts. They were stuck with the alert fatigue issue.

Generation Two

The second generation of security analytics began to incorporate big data and statistical analysis, while remaining a black box to users.

These solutions offered data lakes instead of databases, which allowed for a greater variety of data to be gathered and analyzed, but they were still proprietary. New analytics capabilities emerged, such as the ability to include cloud data, network packets and flow data, but users still couldn’t see how they worked or verify the results.

Data enrichment was better, but users largely could not customize the contextual data they wanted with their alerts. For example, a security team might want to add asset criticality data so they can prioritize events that affect key pieces of their infrastructure or include information from external sources like VirusTotal.

Many solutions started offering threat hunting capabilities as well, which made it easier for security teams to proactively search for suspicious activity that evaded perimeter security controls.

But false positives and limited bandwidth on security teams continued to be a major challenge. In fact, this remains a challenge today. According to the 2021 Insider Threat Report from Cybersecurity Insiders, 33% of respondents said the biggest hurdle to maximizing the value of their SIEM was not having enough resources and 20% said too many false positives.

Generation Three

The third generation of security analytics technologies brings us to the current day, where machine learning, behavioral analysis and customization are driving innovation.

There are now SIEM products that allow organizations to use their existing data lakes, rather than forcing customers to use proprietary ones. And some solutions have opened their analytics, enrichment, and machine learning models so users can better understand them and modify as needed.

Today, powerful algorithms find patterns in data, set baselines and identify outliers. There’s also a greater focus on identifying anomalous behavior (a user taking suspicious actions) and on prioritizing and ranking the risk of alerts based on contextual information like data from Sharepoint or IAM systems. For example, a user accessing source code with legitimate credentials might be a low-priority alert at best, but that user doing so in the middle of the night for the first time in weeks from a suspicious location should trigger a high-priority alert. Thanks to these capabilities, analytic solutions are reaching the point where they can trigger remediation actions automatically.

Security analytics have evolved quickly in recent years and as we look ahead, the industry is starting to combine SIEM, User Entity Behavioral Analytics (UEBA), Security Orchestration, Automation and Response (SOAR) and Extended Detection and Response (XDR) for a more automated and telemetry rich approach to threat detection and response.

But today, the latest advancements are helping to reduce the workload on security teams, allowing them to better detect and contain both known and unknown threats more quickly. Open access to security analytics is also a monumental shift that helps teams better understand and tweak these solutions so they can verify models and generate better results.

Ideally, analytics solutions should have strong pre-built libraries of machine learning models that don’t require users to be data scientists to edit them (but give them the editing option if needed). As these capabilities continue to develop, I believe they’ll be a key factor in helping security teams reduce that 287-day average time to contain a breach in the coming years.