Blog post

Quality and Automation, Partners in Enabling DevOps

By Thomas Murphy | November 06, 2013 | 7 Comments


Recently we have seen a spate of highly visible application roll out failures.  Because of the exposure it seems like these may be “isolated” incidents but I feel they are more normal in large system rollouts then we want to admit.  The core challenge is that generally the business does’t value QA or understand  the “ROI” — “testing just increases the time and cost of delivery”.  This manifests as a constant drive to reduce the cost of testing and if possible avoid certain elements all together.  This persists until somethign blows up or regulatory compliance creates an outside driver.  In general the focus of a company with respect to Quality the focus is on Risk Management (thus risk-based testing methods) rather than Quality as defined by ISO 9126 : Functionality (which includes Security), Reliability, Usability, Efficiency, Maintainability, Portability. 

Quality is important not just from the perspective of “does the application work” but also from the perspective of how will the application impact our maintenance costs?  However, it is very difficult to turn this into an ROI and driving quality can’t just mean triple the testing budget.  A key element is driving automation and getting consistent and continuous. 

Automate everything:

Even if you aren’t “agile” use continuous integration, it drives consistency.  Your CI system should run a consistent set of tests including unit, static analysis, performance, and functional automation.  Static analysis tools provide the ability to detect errors but also to help drive a view of ISO 9126 “conformance”.  Optimyth’s Kiuwan is a great example and you can see on their site what 230 open source projects look like.  I like the way they not only measure quality (5 different components – no one really does 9126) but also provide an estimate of the cost to hit target levels.  Sonar (open source) and the associated commercial offering from SonarSource are becoming increasingly popular due to broad language support and integration to popular CI systems.  There are many compentent static analysis tools to choose from — the key is to make sure you get one, put it to use, build a dashboard and make it visible. 

Functional automation has been a constant challenge for companies.  However pace of deliver, the need to support multiple browsers, devices, geographies will either blow testing costs or be met by automation.  I suggest looking at Google’s Test Blog and their TechTalk’s youtube channel.  Search for GTAC (Google Test Automation Conference) to find a solid set of presentation like this year’s day one keynote: How Facebook Tests Facebook on Android.  If your automation tools and skill set don’t enable you to drive a high degree of automation, you must invest. 

Load and performance testing can’t just happen in a box at the end of the life-cycle.  There should be a level of performance tests that run every day.  Performance testing must also go beyond meeting some pre-defined user load.  Unless you are absolutely sure of the load (and guess what it is a new function, release…people will do things differently than in the “last version”) you have to test to learn.  Test beyond expectation, run failure scenarios (ps, Netflix is another great example to model).  Use Cloud-based load testing that enables you to scale tests to web-scale loads, utilizes real browsers, and enables multiple points of presence.  In other words, make your testing as real as possible.

Build A Production-like Lab

Automate the lab and build one that as closely as possible mimics production.  This may mean the use of virtualization (service, database, network in addition to the lab itself) it also requires, yes, automation.  Use the same ARA tools to release into the test lab that are used in production (ie test release everyday too).  Automate the production of test date either via data generation (ex. Grid-tools VTF) or automated subsetting and masking such as IBM Optim, Informatica Data Subset/Mask, or Grid-Tools. 

Don’t become the next headline, quality matters and if your company wants to move to an agile, continuous delivery world, it is going to need to invest in quality and automation.

Leave a Comment


  • What is the size of DevOps market and expected benefits for test automation organizations?

  • We are working on a new report for sizing the market. We have hesitated to call DevOps a market as it is more about following practicies designed to better coordinate between development and operations. In addition, as with all hot new terms, vendors have been quick DevOps wash their offerings, it would seem that everything is DevOps now. At any rate, we are creating a new document that should publish relatively soon to address this.
    As for benefits for test automation organizations – this will vary as companies go about this in different ways. The key thing I believe will be the ability to have consistent and automated lab environments. We still too frequently have layers of labs that don’t always reflect the production environment and keeping those systems clean and current is a challenge. By employing various practices and tools we have the ability to have more consistent environments.
    The other relation for test automation is that DevOps practices demand a high degree of automation.

  • Satt says:

    Hi Thomas,

    Have you already published the new report on sizing the market?

    Can we give you some inputs from our team’s experience in different market segments.



  • we are always happy for updates

  • Sergio says:

    Hi Mr. Murphy , reading your article i wondered have you heard about Kuscos? Kuscos is a software intelligence platform that provides a range of analysis tools and reports and delivers key information regarding source code modules and team members. Supported languages are Java, JavaScript, C#, VB.Net, VB, Cobol, PL/SQL, Powerbuilder, Delphi and Informix-4GL.

    You can check it on

    Thank you for your time 😉

  • I had not seen that before, please connect with Vendor Briefings if you would like to brief us on this.

  • Mohit Shah says:

    Hi Thomas. I was wondering whether or not the use of Agile & DevOps has impacted QA. Looking at the recent trends, it is the talk of the town.
    What are your thoughts about it?