Now that we are on a subject of testing security and breach/attack simulation tools, one more interesting question arises: if you test security, what constitutes a “pass”? Or, alternatively, at what level do you test?
Think back to the infamous bear analogy. In security, it is NOT a certainty [it is also not obviously false, BTW] that it is enough to outrun the slowest competitor [who the bear can then eat … chomp-chomp], you may actually have to outrun the bear [a Russian APT joke is here somewhere I am sure].
So, there are good arguments that you need to test at the level of your likely adversary [i.e. the above BEAR, even if he is not that FANCY … there!]. But how do you know that level? Even if your threat assessment capabilities are not shitty, this is likely achievable by a tiny minority only… people who basically know what “threat assessment” even means (some actually call it “threat modelling” despite the appsec connotations) …
Here is what I am thinking:
|Conceptual test level||Pros||Cons|
|Test at maximum possible level||You can test whether your security posture will withstand the best of the best||Gap may be so huge, that no clear action plan is evident|
|Test at the level of your current security||Can test the freshly deployed controls; can actually ace the test||Not immediately motivating for making improvements|
|Test at the level of your current security PLUS 10%||Easy to figure out what to improve||How do you actually define and measure it?|
|Test at the level of your past adversary||Can have data, to be turned into test methods||Not a good prediction of the future, perhaps?|
|Test at the level of your likely adversary||Philosophically, this is the best scenario!||Hard to gather solid data on this, and figure it out|
Now, hilarity ensues if “level of your likely adversary” is MUCH higher than “level of your current security” 🙂
Blog posts related to this project: