Gartner Blog Network


Security Testing: At What Level?

by Anton Chuvakin  |  January 29, 2018  |  3 Comments

Now that we are on a subject of testing security and breach/attack simulation tools, one more interesting question arises: if you test security, what constitutes a “pass”? Or, alternatively, at what level do you test?

Think back to the infamous bear analogy. In security, it is NOT a certainty [it is also not obviously false, BTW] that it is enough to outrun the slowest competitor [who the bear can then eat … chomp-chomp], you may actually have to outrun the bear [a Russian APT joke is here somewhere I am sure].

So, there are good arguments that you need to test at the level of your likely adversary [i.e. the above BEAR, even if he is not that FANCY … there!]. But how do you know that level? Even if your threat assessment capabilities are not shitty, this is likely achievable by a tiny minority only… people who basically know what “threat assessment” even means (some actually call it “threat modelling” despite the appsec connotations) …

Here is what I am thinking:

Conceptual test level Pros Cons
Test at maximum possible level You can test whether your security posture will withstand the best of the best Gap may be so huge, that no clear action plan is evident
Test at the level of your current security Can test the freshly deployed controls; can actually ace the test Not immediately motivating for making improvements
Test at the level of your current security PLUS 10% Easy to figure out what to improve How do you actually define and measure it?
Test at the level of your past adversary Can have data, to be turned into test methods Not a good prediction of the future, perhaps?
Test at the level of your likely adversary Philosophically, this is the best scenario! Hard to gather solid data on this, and figure it out

Now, hilarity ensues if “level of your likely adversary” is MUCH higher than “level of your current security” :-)

Ideas? Impressions?

Blog posts related to this project:

Category: security  testing  

Anton Chuvakin
Research VP and Distinguished Analyst
5+ years with Gartner
17 years IT industry

Anton Chuvakin is a Research VP and Distinguished Analyst at Gartner's GTP Security and Risk Management group. Before Mr. Chuvakin joined Gartner, his job responsibilities included security product management, evangelist… Read Full Bio


Thoughts on Security Testing: At What Level?


  1. Nichols says:

    I think the best method that we have today is use frameworks like MITRE Attack and map their techniques to some attacker personas, that have an given behavior and capability and act accordingly some TTPs, resulting in some IOAs (Indicators of Attack). It´s not perfect and will not cover all, but we always have some “Unknown Unknowns” in the threat landscape that will represent some imponderable risks to our organizations that we will need to deal when they arise or are predicted.

    • Thanks for the comment. ATT&CK of course helps A LOT, but the actual test content and the process of running the tests is either on you or on the vendor. A fair amount of work is there…

      Test controls that are meant for UNKNOWN UNKNOWNS is a fun thing, for sure



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.