Recently we have seen a spate of highly visible application roll out failures. Because of the exposure it seems like these may be “isolated” incidents but I feel they are more normal in large system rollouts then we want to admit. The core challenge is that generally the business does’t value QA or understand the “ROI” — “testing just increases the time and cost of delivery”. This manifests as a constant drive to reduce the cost of testing and if possible avoid certain elements all together. This persists until somethign blows up or regulatory compliance creates an outside driver. In general the focus of a company with respect to Quality the focus is on Risk Management (thus risk-based testing methods) rather than Quality as defined by ISO 9126 : Functionality (which includes Security), Reliability, Usability, Efficiency, Maintainability, Portability.
Quality is important not just from the perspective of “does the application work” but also from the perspective of how will the application impact our maintenance costs? However, it is very difficult to turn this into an ROI and driving quality can’t just mean triple the testing budget. A key element is driving automation and getting consistent and continuous.
Even if you aren’t “agile” use continuous integration, it drives consistency. Your CI system should run a consistent set of tests including unit, static analysis, performance, and functional automation. Static analysis tools provide the ability to detect errors but also to help drive a view of ISO 9126 “conformance”. Optimyth’s Kiuwan is a great example and you can see on their site what 230 open source projects look like. I like the way they not only measure quality (5 different components – no one really does 9126) but also provide an estimate of the cost to hit target levels. Sonar (open source) and the associated commercial offering from SonarSource are becoming increasingly popular due to broad language support and integration to popular CI systems. There are many compentent static analysis tools to choose from — the key is to make sure you get one, put it to use, build a dashboard and make it visible.
Functional automation has been a constant challenge for companies. However pace of deliver, the need to support multiple browsers, devices, geographies will either blow testing costs or be met by automation. I suggest looking at Google’s Test Blog and their TechTalk’s youtube channel. Search for GTAC (Google Test Automation Conference) to find a solid set of presentation like this year’s day one keynote: How Facebook Tests Facebook on Android. If your automation tools and skill set don’t enable you to drive a high degree of automation, you must invest.
Load and performance testing can’t just happen in a box at the end of the life-cycle. There should be a level of performance tests that run every day. Performance testing must also go beyond meeting some pre-defined user load. Unless you are absolutely sure of the load (and guess what it is a new function, release…people will do things differently than in the “last version”) you have to test to learn. Test beyond expectation, run failure scenarios (ps, Netflix is another great example to model). Use Cloud-based load testing that enables you to scale tests to web-scale loads, utilizes real browsers, and enables multiple points of presence. In other words, make your testing as real as possible.
Build A Production-like Lab
Automate the lab and build one that as closely as possible mimics production. This may mean the use of virtualization (service, database, network in addition to the lab itself) it also requires, yes, automation. Use the same ARA tools to release into the test lab that are used in production (ie test release everyday too). Automate the production of test date either via data generation (ex. Grid-tools VTF) or automated subsetting and masking such as IBM Optim, Informatica Data Subset/Mask, or Grid-Tools.
Don’t become the next headline, quality matters and if your company wants to move to an agile, continuous delivery world, it is going to need to invest in quality and automation.
View Free, Relevant Gartner Research
Gartner's research helps you cut through the complexity and deliver the knowledge you need to make the right decisions quickly, and with confidence.Read Free Gartner Research
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.