The evolution of security metrics for NIST CSF 2.0

CISOs have long been spreadsheet aficionados, soaking up metrics and using them as KPIs for security progress. These metrics have traditionally measured specific systems or single indicators — vulnerabilities detected, percentage of vulnerabilities patched, software and hardware asset inventory coverage, etc. The NIST Cybersecurity Framework (CSF) 2.0 underscored that metrics like these alone are insufficient and probably even improper when used as proxies for security outcomes.

Siloed, narrow metrics do have a place in cybersecurity, of course. However, the intelligent CISO will consider these standalone metrics as only one type of information — a type that is useful, but can also be inexact or be gamed. For that reason, CISOs that want a complete picture must also closely examine security processes.

Combining effective use of metrics plus a deeper understanding of how security processes play out is the best way to build more security agility and enable teams to react more quickly and effectively.

CISOs need a new set of metrics around process compliance to provide observability into how teams perform their work. Process metrics can span multiple systems and attributes, breaking down silos and providing a more holistic view of security that more closely aligns with CSF 2.0’s emphasis on outcomes over old-style metrics that measure a single attribute or factor.

Why CISOs fell in love with spreadsheets and metrics

In the beginning, CISOs got siloed reports in spreadsheets, usually one from each tool. Analysts would manually aggregate them into something valuable and comprehensive. As visual presentation layers improved and security tool integrations via APIs became more common, CISOs caught the metrics bug and started building dashboards to complement their spreadsheets. And why not? Who doesn’t love a crisp, well-designed system for laying out statistics and communicating stories and ground truths that run on autopilot? In many cases, the dashboard was an extension of their original spreadsheets. The dashboard, like spreadsheets, also offered the promise of KPI compatibility and adaptability.

Accountability has always been a core challenge for CISOs. Dashboards and metrics put the icing on the accountability layer cake, making it easy and eye-catching. In cybersecurity, the allure of clear, quantifiable metrics is undeniable. Dashboards gleam with the promise of control, offering a seemingly straightforward way to gauge an organization’s cyber health. Metrics could be malleable, helping CISOs quickly tell stories about progress or create rationales for further investment.

Also, critical dashboards and metrics offered a way to simplify managing up. In board meetings and weekly ELT confabs, the CISO could send a link to the latest dashboard alongside a paragraph of commentary and call it a day. After all, every other C-team leader had their metrics, from cost of capital and cost per headcount to recurrent revenue and marketing qualified leads. Security metrics, painted in a dashboard, would drop seamlessly into the deck. Or you can export it to a spreadsheet for the finance team, who speaks spreadsheets.

Why CISOs must equal weight metrics and process

Yet, beneath this veneer of certainty lies a complex truth: individual metrics, while helpful, are fundamentally limited in that they only measure singular data points from siloed systems. More progressive CISOs are taking a more complex look at their dashboards and putting a magnifying glass to the processes affecting metrics and outcomes. As Bruce Schneier has long argued, security is a process — not a product.

To understand why process metrics are a necessary complement to traditional siloed metrics, consider the recent attacks by Chinese and Russian hackers that allowed them to access email accounts of top US government officials and top Microsoft brass, respectively.

Those attacks combined crude methods with clever social engineering and, in some cases, exploitation of known vulnerabilities that were not considered extremely high risk. In some cases, the attackers targeted legacy internal servers that had remained connected to corporate networks in error.

These attacks would have largely flown under the radar of security dashboards. Analysis of these incidents faulted primarily the security culture of the offending organizations — in other words, lack of or improper security processes. Instances of weak or failed processes included insufficient checks to ensure legacy systems were offline and inadequate certificate rotation processes to head off persistent attacks. The bottom line?Metrics alone wouldn’t tell this story of a broken process. More broadly, a narrow reliance on single data points, not metrics, as part of a more extensive process is dangerous for CISOs.

• Siloed metrics don’t tell the whole story: Narrow metrics may show the number of vulnerabilities patched but don’t capture root causes, how those vulnerabilities were prioritized, or if the most critical ones were addressed first. Narrow metric measurements alone won’t reveal if teams are following consistent patching protocols or if they’re making ad-hoc decisions.
• Siloed metrics are easily gamed: If teams are held accountable purely by metrics, they may focus on hitting targets rather than achieving true security. Inflated results or cherry-picked statistics can be curated to create a false sense of accomplishment instead of driving meaningful improvements.
• Siloed metrics don’t reflect adaptability: Focusing solely on metrics can stifle innovation and preparedness. Siloed metrics often reflect historical data, not the speed or effectiveness of response to a newly emerging threat.

None of this is to say narrow metrics are useless. Their directionality can hint at larger problems or process weaknesses. They are an excellent way to snapshot a state of readiness of security controls or the volume and type of attacks, for example.

Narrow metrics are a way to measure the day-to-day security posture of an organization. But by widening the focus from narrow metrics to process metrics, CISOs can achieve better security outcomes. Here are some advantages of adding process analysis to the mix:

• Holistic understanding: Process-driven security provides a comprehensive understanding of how security measures work together as part of larger processes. This understanding allows CISOs to answer complex questions, make informed decisions, and identify areas for improvement.
• True accountability: Prioritizing process over metrics ensures that accountability is not solely based on hitting targets but on achieving meaningful security improvements. This approach prevents teams from gaming metrics and encourages a focus on actual security outcomes.
• Adaptability and innovation: Metrics often reflect historical data and may not capture the speed and effectiveness of response to emerging threats. Process prioritization encourages adaptability and innovation, enabling organizations to stay ahead of evolving cyber threats.

How CISOs can build a better process muscle

Building a process-driven security approach is more complex than instituting a slate of checklists and playbooks (although those have their place). Here are some basic steps to build a better process-driven culture and approach.

Analyze and instrument security processes. What is not measured, cannot be improved on. Deploy methods to trace and log security processes to enable systematic analysis of process failures and gaps. Without frequent analysis and retrospectives, teams will not learn from their actions, and collective learning will not scale up through shared learnings. Ironically, this creates another form of metrics — but with a difference. Process metrics look at what systems and people do and how they do it, using live data directly from your environment. It is a lens into the human element of security posture management and response.

Provide process guidance, not rules. Processes become more powerful when teams are allowed to adapt them to their own needs and situations. CISOs who want to unlock the wisdom and power of their security teams will give them guidance on the outcomes they desire rather than specific rules on how to achieve those outcomes.

Let human experts lead the way. The best security engineers are geniuses at pattern recognition. Their logic in root-causing attacks often provides excellent guidance for designing processes. This also gives the teams agency and lets them create and adapt processes that match their workflows.

Today’s CISOs face a dizzying array of cybersecurity challenges. CSF 2.0 and its movement toward outcomes and processes over metrics acknowledges that the old reliance on simple metrics and dashboards was insufficient to meet the rapidly evolving threat landscape.

Capturing processes with compound metrics that span systems and show behaviors and events in the aggregate will help drive security teams to focus on outcomes and to become more adaptable by gaining a clear vision of how they behave when it matters the most.