Gartner Blog Network


Things That Aren’t Risk Assessments

by Ben Tomhave  |  July 24, 2014  |  2 Comments

In my ongoing battle against the misuse of the term “risk,” I wanted to spend a little time here pontificating on various activities that ARE NOT “risk assessments.” We all too often hear just about every scan or questionnaire described as a “risk assessment,” and yet when you get down to it, they’re not.

As a quick refresher, to assess risk, you need to be looking at no less than 3 things: business impact, threat actor(s), and weaknesses/vulnerability. The FAIR definition talks about risk as being “the probable frequency and probable magnitude of future loss.” That “probably frequency” phrase translates to Loss Event Frequency, which is compromised of estimates that a threat community or threat actor will move against your org, that they’ll have a certain level of capabilities/competency, and that your environment will be resistant to only a certain level of attacks (thus representing weakness or vulnerability).

Oftentimes, that “probable magnitude” component is what is most lacking from alleged “risk-based” discussions. And, of course, this is where some of these inaptly described tools and capabilities come into play…

Questionnaires

A questionnaire is just a data gathering tool. You still have to perform analysis on the data gathered, supplying context like business impact, risk tolerance/capacity/appetite, etc, etc, etc. Even better, the types of questions asked may result in this tool being nothing more than an audit or compliance activity that has very little at all to do with “risk.” While I realize that pretty much all the GRC platforms in the world refer to their questionnaires as “risk assessments,” please bear in mind that this is an incorrect characterization of a data gathering tool.

Audits

The purpose of an audit is to measure actual performance against a desired performance. Oftentimes, audits end up coming to us in the form of questionnaires. Rarely, if ever, do audits look at business impact. And, one could argue that this is ok because they’re really not charged with measuring risk. However, we need to be very careful about how we handle and communicate audit results. If your auditors come back to you and start flinging the word “risk” around in their report, challenge them on it (hard!!!) because dollars-to-donuts, they probably didn’t do any sort of business impact assessment, nor are they even remotely in the know on the business’s risk tolerance, etc.

Vulnerability Scans and AppSec Testing

My favorite whipping-boy for “risks that aren’t risks” is vulnerability scans. I had a conversation with a client last week where they had a very large (nearly 100-page) report dropped onto them that was allegedly a “risk assessment,” but in reality was a very poor copy-n-paste of vuln scan data into a template that had several pages of preamble on methodology, several pages of generic explanatory notes at the end, and did not appear to do any manual validation of findings (the list of findings also wasn’t deduplicated).

Despite the frequent use of “risk” on these reports, they’re most often describe “ease of exploit” or “likelihood of exploit.” However, even then, their ability to estimate likelihood is pretty much nonsensical. Take, for instance, an internal pentest of a walled-off environment. They find an open port hosting a service/app that may be vulnerable to an attack that could lead to remove root/admin and is easy to exploit. Is that automatically a “high likelihood” finding? Better yet, is it a “high risk” finding? It’s hard to say (though likely “no”), but without important context, they shouldn’t be saying anything at all about it.

Of course, the big challenge for the scanning vendors has always been how to represent their findings in a way that’s readily prioritized. This is why CVSS scores have been so heavily leveraged over the years. However, even then, CVSS does not generally take into account context, and it absolutely, positively is NOT “risk.” So, again, while potentially useful, without context and business impact, it’s still just a data point.


I could rant endlessly about some of these things, but I’ll stop here and call it a day. My exhortation to you is to challenge the use of the word “risk” in conversations and reports and demand that different language be used when it’s discovered that “risk” is not actually being described. Proper language is important, especially when our jobs are on the line.

Category: risk-management  

Tags: assessment  grc  language  management  risk  terminology  

Ben Tomhave
Research Director
1 years at Gartner
19 years IT Industry

Ben is conducting research within the Security and Risk Management Strategies team under Gartner for Technical Professionals.


Thoughts on Things That Aren’t Risk Assessments


  1. Mike Fitzpatrick says:

    Nice article. Wish more folks had this perception.

  2. Nurudeen Odeshina says:

    Great thoughts and thanks for putting this into context. I get puzzled too as regards the Vulnerability Scans and AppSec Testing. Imagine performing a random, enterprise scan and obtaining a CVSS critical/high score on a system not critical to an organization and shouting “high risk”. This would happen if context and business impact are not put into perspective from the start. A great way to start is to understanding the context (critical systems) and appropriate definition of impact parameters as well as other parameters.



Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.