Gartner Blog Network


The Summer of Testing Research

by Thomas Murphy  |  April 7, 2015  |  5 Comments

We have kicked off our next round of research for the software testing tools market.  We are close to publication on a new Market Guide for API Testing and Service Virtualization and are going to continue expanding the notes we have to cover the overall market.  In the past we have published a Magic Quadrant for Integrated Software Quality Suites.  This document was designed to give a view of the enterprise ready providers of suites of tools designed to cover “all” of your testing needs.  It was initially focused on Test Management, Functional Test Automation, and Performance Testing.  Over time we included elements like SOA testing, code quality, unit testing, etc. into the elements of what makes up the Breadth of Vision axis.

Now, with both an expanded team and an expanding variety of software to test, we are going to expand the number of documents we produce as well as shifting and tightening the focus of the MQ.  The MQ will change from being Integrated Software Quality Suites to the MQ for Software Test Automation.  Thus is will focus on functional test automation and be targeted primarily at GUI layer test automation (API/Service level testing being covered in the soon to be released Market Guide).  We will also produce a critical capabilities document for this market and while still tuning the use cases are looking at: Package Application Testing (the back office), Responsive Web (the new desktop), and Mobile (field and customer).  We believe this will better address the inquiry flow that we typically have than the current document.

In conjunction with this we will also take the major pieces removed from the current MQ: Peformance/Load testing and test  management and move these into separate Market Guides.  Thus by the end of the year we will have four Market Guides:

  1. Mobile/Device Management
  2. API Testing and Service Virtualization
  3. Performance Testing
  4. Test Planning and Management

as well as the restructured Magic Quadrant for Software Test Automation and a Critical Capabilities for Test Automation.

We are currently in the kick-off phase updating our vendor/product lists, firming up the use cases and designing the surveys we will use for vendors and users.  We are also looking to produce updates and/or new documents on test automation, QA team structures (testing and bimodal IT and agile/devops), and coordinate with our security and UX teams around the broader view of software quality.  If there are specific themes or issues you would like to see addressed in this research please either leave responses to this doc, connect to us for a client inquiry or message me on twitter (@metamurph).

Note that our documents on tools are focused on commercial companies as Magic Quadrants illustrate the capabilities of different vendors to service the needs of enterprise IT organizations.  Thus open source will only appear in lists or where it has a commercial offering and we are evaluating the provider. Our goal is to complete the majority of this work by the end of Summer.

Category: devops  loadstress-testing  mobile  mq  quality  testing  usability-testing  

Thomas E. Murphy
Research Director
15 years at Gartner
33 years IT industry

Thomas Murphy is a research director with Gartner, where he is part of the Application Strategies and Governance group. Mr. Murphy has more than 33 years of experience in IT as a developer, product manager, technical editor and industry analyst. Read Full Bio


Thoughts on The Summer of Testing Research


  1. Hi Tom
    great move to break out the MQ & research into these separate areas. Looking forward to collaborating on the research.
    Sandeep

  2. Hi Tom,

    Sounds like you’ve got quite a summer planned :) I agree with Sandeep here….good call on breaking out the MQ. We’re happy to help in any way as you pull all this together.

    -Jonathan

  3. Udhay says:

    This is just Superb. Crystal clear on interpreting the MQ. The reach is more now. Should we call it as Performance Testing and Engineering? And, instead of Automation as SDET (Software Design Engineering in Test)

  4. You are correct that Performance is shifting to an engineering focus but the tools are still performance testing centered. I am sure we will have something specific to performance engineering before the end of the year.

  5. Matthias Rasking says:

    Sounds exciting and like the right way to go. Especially interested in the QA team structures work, DevOps is changing a lot of thinking in this area although some industries still struggle with testing as an independent team and now face challenges with time-to-market given the digital disruption of their businesses.



Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.