I see some strange proposals for security metrics. Individually of course every proposal carries meaning for the proposer so one shouldn’t criticize too harshly. But authors of metrics do need to remember that there’s no point measuring something unless you’re going to do something about it – and that means you need to measure that thing in the right way. The most common issues involve counting numbers, presenting percentages, targeting totals and trends.
Numbers mean quantification, right? Yes, but they’re USELESS ON THEIR OWN. We have AV on 358 systems. Great. But how many systems are there? Has that number grown or shrunk since last week? Please, don’t just – all you’ll end up with is a number of widgets that your leaders won’t understand – and in the worst case could drive absolutely the wrong response.
A % sign helps. We have AV on 93% of systems is much more useful than ‘358’. But you have to bound the denominator sometimes. 93% of what? Endpoint systems? Servers? The population of target environments for any control may have sub populations (think about critical production systems vs development environments, or PII data vs the canteen menu), and you must be clear what the actual target population is.
Trends are often more important than absolutes. Yes I care that half our firewall rules haven’t got comments, but I really want to know that we’re making it better. Trends also help when you can’t control something, but it influences what you do – number of spam emails, incidents, malware detection, and so on. If the number is going in the wrong direction you need to change something – your tools, processes or resources. If it’s going in the right direction, then double down on what you’re doing.
Targets can be evil. Someone has to decide what the right number is – how? What is the right target for number of rules in a firewall? How long should it take us to detect an attack? Targets can be unintentionally un-achievable – which can have really bad outcomes. Targets also drive behavior – the law of unintended side effects. Think really carefully before making something a target, and make sure you have a balanced approach and message to the workforce.
Security metrics are really important. They tell us how well we are doing, help us prioritize resources, and support communication to leadership. But take care in how you enumerate them – random metrics really don’t help.
A much more in-depth research note on this (with a whole load of example security metrics included) is Developing Metrics for Security Operational Performance. (Gartner license is required.)
View Free, Relevant Gartner Research
Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.Read Free Gartner Research
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.