Gartner Blog Network


Numbers, Percentages, Targets and Trends (Security Metrics gone wrong)

by Mike Wonham  |  April 2, 2020  |  Submit a Comment

I see some strange proposals for security metrics. Individually of course every proposal carries meaning for the proposer so one shouldn’t criticize too harshly. But authors of metrics do need to remember that there’s no point measuring something unless you’re going to do something about it – and that means you need to measure that thing in the right way. The most common issues involve counting numbers, presenting percentages, targeting totals and trends.

Numbers

Numbers mean quantification, right? Yes, but they’re USELESS ON THEIR OWN. We have AV on 358 systems. Great. But how many systems are there? Has that number grown or shrunk since last week? Please, don’t just – all you’ll end up with is a number of widgets that your leaders won’t understand – and in the worst case could drive absolutely the wrong response.

Percentages

A % sign helps. We have AV on 93% of systems is much more useful than ‘358’. But you have to bound the denominator sometimes. 93% of what? Endpoint systems? Servers? The population of target environments for any control may have sub populations (think about critical production systems vs development environments, or PII data vs the canteen menu), and you must be clear what the actual target population is.

Trends

Trends are often more important than absolutes. Yes I care that half our firewall rules haven’t got comments, but I really want to know that we’re making it better. Trends also help when you can’t control something, but it influences what you do – number of spam emails, incidents, malware detection, and so on. If the number is going in the wrong direction you need to change something – your tools, processes or resources. If it’s going in the right direction, then double down on what you’re doing.

Targets

Targets can be evil. Someone has to decide what the right number is – how? What is the right target for number of rules in a firewall? How long should it take us to detect an attack? Targets can be unintentionally un-achievable – which can have really bad outcomes. Targets also drive behavior – the law of unintended side effects. Think really carefully before making something a target, and make sure you have a balanced approach and message to the workforce.

Security metrics are really important. They tell us how well we are doing, help us prioritize resources, and support communication to leadership. But take care in how you enumerate them – random metrics really don’t help.

A much more in-depth research note on this (with a whole load of example security metrics included) is Developing Metrics for Security Operational Performance. (Gartner license is required.)

 

 

Additional Resources

View Free, Relevant Gartner Research

Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.

Read Free Gartner Research

Category: security-operations  security-operations-for-technical-professionals  

Mike Wonham
Sr Director Analyst I
3 years at Gartner
22 years IT Industry

Mike Wonham works in GTP Security, Identity and Risk, focusing on data security, classification, DLP, risk process and measurement, security governance and programs and security metrics. Read Full Bio




Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.