Neil MacDonald

A member of the Gartner Blog Network

Neil MacDonald
VP & Gartner Fellow
15 years at Gartner
25 years IT industry

Neil MacDonald is a vice president, distinguished analyst and Gartner Fellow in Gartner Research. Mr. MacDonald is a member of Gartner's information security and privacy research team, focusing on operating system and application-level security strategies. Specific research areas include Windows security…Read Full Bio

Coverage Areas:

Are Web Application Security Testing Tools a Waste of Time and Money?

by Neil MacDonald  |  August 25, 2009  |  15 Comments

My previous post on the value of linking web application vulnerability scanning tools with web application firewalls generated a lot of discussion. Take a look through the post and the lengthy comment string.

Let me state up front that I firmly believe we should change our development processes (and developer culture) to produce more secure code. This applies to outsourced development as well. This should be our first priority. A tool cannot solve what fundamentally is a process problem. A tool can help us to scale the process, make testing more efficient and to make it repeatable, but we must address the process problem first.

Web application security testing tools (a form of dynamic application security testing [DAST] tools) test a web-enabled application in its assembled and running state looking for conditions that would be indicative of a security vulnerability. For example, by trying to inject SQL into user-input fields, a DAST scanner might see responses from the application that indicate vulnerability to SQL injection.

The market for DAST tools is about US $100M and growing. Larger vendors such as IBM and HP have made their acquisitions in this space and have integrated the capability into their development platforms. In addition, DAST tools are available from independent commercial vendors such as Accunetix, Cenzic, and NT Objectives .Open source alternatives are available as well – Nikto and OWASP’s WebScarab are mentioned most frequently.

Are DAST tools a waste of time and money? I don’t believe so. Despite our best efforts to produce and procure more secure code, applications will still have vulnerabilities. Rather than wait on the bad guys to find our vulnerabilities for us, DAST tools in the hands of a skilled professional can help identify vulnerabilities that we can then fix (or shield). Even if DAST tools (or static application security testing tools for that matter) aren’t perfect (false positives and false negatives), using them is better than not using them. In other words, a flashlight in a dark room is better than being completely in the dark.

What do you think?

15 Comments »

Category: Application Security     Tags: ,

15 responses so far ↓

  • 1 MikeA   August 25, 2009 at 4:04 pm

    They are useful, to a point. I think McGraw said it best by introducing the term “badness-ometer”

    http://www.cigital.com/justiceleague/2007/03/19/badness-ometers-are-good-do-you-own-one/

    The danger is when the badness-ometer (DAST) says you are “clean”. At that point, do you believe them, or is there more to do. I belive we know what the answer is, but many people feel that passing the badness-ometer is the only hurdle.

  • 2 Neil MacDonald   August 25, 2009 at 6:02 pm

    @MikeA,

    I like it and I agree. But it’s not just web application scanners. Nothing can tell me with 100% certaintiy if I am secure – not a DAST tool, not a SAST tool, not a full penetration test. All I can do is get more evidence that supports the “trustability” of the application.

    Each organization needs to decide for themselves based on the application, the data it holds, its exposure to outside users, etc etc what level of “trustability” (my word) or the inverse (badness-ometer) is required.

    In no way should a clean web application scan be interpreted as the application is vulnerability-free. It just means the tool didn’t find anything…

    SAST raises the bar.

    Pen-testing with some manual code review takes it even further.

    Each one takes more time and costs more, but finds more.

    Each organization needs to decide where on this cost/benefit/security curve they are comfortable and it is *not* the same for every application.

  • 3 Twitter Trackbacks for Are Web Application Security Testing Tools a Waste of Time and Money? [gartner.com] on Topsy.com   August 26, 2009 at 1:37 am

    [...] Are Web Application Security Testing Tools a Waste of Time and Money? blogs.gartner.com/neil_macdonald/2009/08/25/are-web-application-security-testing-tools-a-waste-of-time-and-money – view page – cached [...]

  • 4 James T   August 26, 2009 at 9:45 am

    This isn’t about a single item that will “make you secure”. Web Application Scanning should be part of an overall security program. They do allow you to identify vulnerabilities that one might not know existed otherwise. But this can’t be done once. Applications, and the environment they run in, change over time. The attacks change over time as well. For this reason, application assessments should be done on recurring basis.

    Further, as Neil mentions, the Development and QA processes need to change to include security type of tests in the standard QA test suites. Not as a onetime step tacked on to the end of a development cycle. This is the process part. There is also a fundamental knowledge issue where a lot of developers still do not understand or think about the security implication of their code. Training and continued focus on the security for, and by the developers, is an essential part of an overall security of the application.

    The testing, either source code testing done during the development process, or the DAST type of testing done in staging or production is only a “check” not a mitigating control.

    Cheers!

  • 5 Jim’s Bloggyness » Post Topic » An Information Security Place Podcast – Episode #24   September 3, 2009 at 12:46 am

    [...] Topic – Web App Scanners And Web App Firewalls According to Gartner – Link 1 / Link [...]

  • 6 An Information Security Place Podcast » Blog Archive » An Information Security Place Podcast – Episode 24   September 3, 2009 at 9:09 am

    [...] Topic – Web App Scanners And Web App Firewalls According to Gartner – Link 1 / Link [...]

  • 7 An Information Security Place » An Information Security Place Podcast – Episode 24   September 3, 2009 at 9:10 am

    [...] Link 1 / Link [...]

  • 8 Mario Lacroix   September 10, 2009 at 12:59 pm

    I agree, but have some points to add.

    DAST/SAST tools are a great way to reduce outside attacks, as most of these attacks are based exactly in the same rules the tools are made for (ie.: SQLi, XSS, XSF, …).

    Unfortunately, we have another threat that come from “inside the company” (example: the system analyst or the developer himself) that knows many more about the software (and the systems that hosts it) them anyone… Of course, an attack that came from these guys can’t be stimulated or reproduced with no tool…

    My point is: If you can trust your people and apply a secured SDLC, secure coatch, culture, etc., the use of a DAST/SAST tool can help a lot the production of secure code… but if you didn’t, do your homework first or no tool can help you.

    Another thing is: If your company likes to use outsource staff to develop, consider pay more for a “trusted” (or at least a big) company, as the have CMM, and well defined process to develop and review the code. Please, eliminate the “small companies” (one-team guy) from your process to develop sensible software or software that connects to sensible systems… Otherwise, you can find some Easter eggs where you won’t like to.

    Thanks a lot,

    Mario Lacroix

  • 9 Manickam K   September 16, 2009 at 2:28 am

    Web Application Security Testing Tools helps identifying at least known vulnerabilities and also helps to review the status of the identified vulnerabilities once the same is fixed. If one is relying on Web Application Security Testing Tools to certify web application security, then that is not accurate. Web application security can be achieved by addressing the security requirements at various stages of SDLC. From Architecture & design review, secure coding, Code review, Security testing on top of QA, Automated Web Application Security Testing Tools and Manual review.

  • 10 Neil MacDonald   September 16, 2009 at 11:56 pm

    @Manickam,

    Agree that web application security testing tools can’t prove an application is secure. But, neither can the full SDLC process changes. There is no silver bullet.

    Given the choice between doing nothing and only doing web application security testing after-the-fact, I’d take the testing.

    Given the choice between doing web application security testing after-the-fact and doing the full SDLC process changes you describe, there is no doubt I’d go with the full SDLC process change.

    The latter gives a higher degree of “trustability” in the application…

  • 11 Jaime Castells   October 9, 2009 at 12:34 pm

    I take exception to one comment in the article: “…we must address the process problem first.” I feel this misses an opportunity.

    Current generation DAST have arespectably low false positive rate and do, as the article points out, provide efficiency and repeatability in the testing process. As a result, they can be used to produce reasonbly high quality metrics. Metrics are an excellent tool to drive change.

    The problem most security practitioners face is slow adoption which, at least in part, results from weak management buy-in. In trying to build a secure application practice or to improve the maturity of a development process, it makes great sense to start with something that can produce straight forward metrics that can be redily communicated to management.

    I believe that incorporating DAST early in the growth of a SSDL can help to demonstrate value and improve the chances of long term success.

  • 12 Neil MacDonald   October 10, 2009 at 10:30 am

    @Jaime,

    Thanks for reminding us of this reality. I believe these are actually parallel efforts. We must acknowledge, understand and put into motion efforts to produce and procure more secure applications. At the same time, we have inherited a legacy portfolio of insecure apps in production that are very likely insecure. These must be tested as well, ASAP especially for those that are externally accessible and host sensitive data. Its likely the security organization that will take on this responsibility and DAST tools are a way to do this.

    To take this further, if the DAST tools find a vulnerability in an already-deployed app, what do you do? That’s where the discussion of WAFs as a short-term shield came in:

    http://blogs.gartner.com/neil_macdonald/2009/08/19/security-no-brainer-9-application-vulnerability-scanners-should-communicate-with-application-firewalls/

    That created a lot of discussion, but I remain convinced of the value of WAFs for this (and a few others that are in the comments) scenario.

  • 13 Glenn Gramling   November 28, 2009 at 4:59 pm

    What do you think the adoption rate of DAST tools will be in 2010/11? Given the market is $100M, do you see adoption for both market growth and user acceptance? What do you think has limited the growth to date (beyond weak management buy in)?

  • 14 didier   December 22, 2009 at 10:01 am

    I agree with the article and we had a very good experience with a online vulnerabilities scan from http://www.gamasec.com that provide deep technical and executive summary reports with recommendations.

    So for us the solution of havinf a tools that provide us with monthly reports and defirential report is a great security issue.

    D

  • 15 john doeuf   December 22, 2009 at 10:05 am

    http://www.gamasec.com was one of the scanner that we chalenge and we were satisfy with the result so we are now using this website security tools for our website scan on a monthly basis

    JD