As I alluded here, we [Augusto and me] will be starting an epic new research project on testing security [BTW, should we codename it “Testing Security”, Augusto? :-)]
First, a quick poll: how many types of security testing do you know? Let me try…
- Penetration testing (PT)
- Red teaming (RT) – differences of PT and RT are discussed here
- Vulnerability assessment
- Application security testing (AST)
- Security rating services (like BitSight and SecurityScorecard)
- Attack, threat, breach simulation tools (details)
- Others, possibly many others….
Now, at the risk of sounding too philosophical, what does it even mean “to test one’s security”? At this early stage of our effort, all bets are off, but we do want to look into testing of security technologies and processes (such as detection and response processes), with the focus on outcomes, not controls. In essence, we want to look into testing the effectiveness, not the presence, of security controls.
Please share how you think of TESTING SECURITY … and please don’t say that “a good pentest should be enough security testing for everybody.” 🙂
Note that we will try to keep the focus of this on actually TESTING, not other forms of evaluating, measuring, assessing or guessing about security … Asking people how secure they think they are isn’t testing. Questionnaires isn’t testing. Reviewing policies isn’t testing. Even reviewing the lists of controls they think they deployed isn’t testing (IMHO).
In fact, overall, paper security isn’t.
Blog posts related to Testing Security project:
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.
Comments are closed
17 Comments
One of my favorite meaning about “testing” is “experimentation”, which in turn means “scientific method that, starting from a hypothesis, consists in the observation and classification of a phenomenon under controlled conditions.” In a lab we have that “controlled conditions” to use any testing method to observe if a solution will or will not behave as it is promises.
The hard fact is that on reality we have the imponderable variable of the human adversary, that changes all the game mechanics. So, independent of the testing methodology we need to bring the attacker mindset/behavior to the chessboard, as close as possible to simulate to assess if a solution really improve security or it´s just another electron consumer that occupies 6U on the Datacenter….
Thanks a lot for the insightful comment. Indeed, if we TEST SECURITY **well**, we imply that said test is somehow testing your security readiness for a real attacker (’cause if it does not, then it kinda has not real point). However, HOW to achieve such “real attacker similarity” is a matter of much debate…..
The topics relates to security metrics, I believe.
Thanks for comment, but this is tricky. We also think that this topic is **related** to metrics, but it is not the same as metrics. We think you can measure and then product metrics, but it may or may not constitute a test. You can measure something useless (like how many firewalls are purchased or viruses caught), and this would be a metric but not a test…
Would you be including active audits, or purposely avoiding the audit baggage? 🙂
I also first thought about this as being like QA for security controls. Testing against expected outcomes, load, unexpected input, blind spots.
Thanks for the comment. OK, I will bite.
In your learned opinion, do **IT audits** actually test security?
About your other point, indeed, “QA for security control” or even (perhaps?) “QA for security programs” is a useful way to think about it.
I think I would personally keep “audit” and “test” as separate deals. I can certainly consider overlap there, but ultimately I think both functions could stand on their own.
Thanks for the comment. Indeed, I try to stay away from the word “AUDIT” since too much grief is associated with it 🙁
Adrian Sanabria alluded to some of this in an old peerlyst article – “Due diligence isn’t just something you do before buying a product – it needs to done after implementation and throughout the product lifecycle.” I think pen testing and red teams are good, but a point-in-time approach. It’s hard to expect a human to try data exfil via DNS tunneling (as an example) 365 days a year. Our environment is so dynamic now.. Therefore: (1) We must be able to validate continuously and automatically (2) with a true attacker mindset (i.e utilizing all techniques not just vulnerabilities) (3) AND be safe while we are validating so we can run tests continuously and automatically in a real production environment to again best replicate the attacker.
Agreeing with the first comment that the security community in general needs to embrace the attacker mindset. It’s not about what security controls we are deploying, it’s how effectively they work against attackers.
Thanks a lot for the comment. I think both PT/RT and threat simulation would be big parts of it, but I suspect there are others as well…
Just adding a link to our perspective:
https://safebreach.com/Post/Breach-and-Attack-Simulation-Versus-Pen-Testing-and-Red-Teaming
Thanks for the comment!
You make a good point, but I feel Pen Testing is a good starting point. In an active network, we all know it’s all too easy for one team to ask for a port to be opened or for an account password to be left vulnerable. It’s important to regularly ask questions and we always say you can’t mark your own homework.
A firm that takes it seriously will just start with the Pen test and follow up with professionals who’ll think more laterally.
Oh, for sure! I fully agree that PT is the best 1st answer for most, but I don’t think it is the last answer. PT is periodic, results are sometimes (often?) ignored, etc. IMHO, PT is a good test to prove that your security is shit, but perhaps not the best way to prove that it is great….
I had a lot of success around targeted testing of specific products as they’re being evaluated and deployed in the organization. Rather than operational testing (such as pen testing) these tests focus on the quality and the security controls of systems helping make better configuration and deployment decisions. These tests also feed pen test activities by jumpstarting the knowledge about the organization. These targeted tests live (I think) between your AST and pen tests.
I gave a talk on “When to Test and How to Test It” at DerbyCon that covers a fair bit of this ground. Might be useful. http://www.irongeek.com/i.php?page=videos/derbycon7/t300-when-to-test-and-how-to-test-it-bruce-potter Caution: non-trivial amount of f-bombs in that talk.
Thanks a lot for your insightful comment. We DO plan to cover security control/tech testing and not just testing the entire org security. In our experience “does this shit even work?” applies equally to security tools and processes alike 🙂
Thanks for the video.
And, finally, our early research discovery was that discussing this topic without f-bombs is very difficult if not impossible 🙂
I’ve generally referred to this as “product due diligence” assessments and I agree it’s a separate category from anything else Anton mentioned above. It’s a different process from pentesting, red teaming, controls testing or anything else mentioned. The threat mapping chapter in NoStarch’s recent Car Hacking book is an excellent example of this.