Jonah Kowall

A member of the Gartner Blog Network

Jonah Kowall
Research Vice President
3.5 years with Gartner
20 years IT industry

Jonah Kowall is a research Vice President in Gartner's IT Operations Research group. He focuses on application performance monitoring (APM), Unified Monitoring, Network Performance Monitoring and Diagnostics (NPMD), Infrastructure Performance Monitoring (IPM), IT Operations Analytics (ITOA), and general application and infrastructure availability and performance monitoring technologies. Read Full Bio

Coverage Areas:

Criteria for the 2014 Magic Quadrant for Application Performance Monitoring

by Jonah Kowall  |  May 29, 2014  |  10 Comments

Gartner is beginning the research process for the 2014 Magic Quadrant for Application Performance Monitoring. Will Cappelli and myself will be running the research as we have for the last few years. The release is currently planned for the 4th quarter of 2014, and we will be sending out surveys to vendors next week. Based on the data below, if you believe you qualify for the research please get in touch with myself via email or Twitter (@jkowall):

The 2013 Magic Quadrant saw minor changes around the weighting and importance of analytics and mobile. The 2014 research once again sees importance in these two critical trends, but also a focusing on SaaS delivery being essential as enterprises continue to adopt various public cloud delivered applications, services, and infrastructure.

Market Definition Is Unchanged

In the 2014 Magic Quadrant, the market definition will remain unchanged from the 2013 version (see “Magic Quadrant for Application Performance Monitoring”). While we have made changes in the application definition. Gartner defines an application as a software program or group of programs that interact with their environment via defined interfaces and which are designed to perform a specific range of functions. They may be end user facing, presenting a user interface, or provide the interface between two applications themselves. Applications are not (normally) wholly independent as they typically require an operating system or multiple operating system instances to manage their use of physical and logical computing, storage and network resources within a data center, or provided by 3rd parties.

Gartner defines APM as having five dimensions of functionality:

  1. End-user experience monitoring (EUM) — The capture of data about how end-to-end latency, execution correctness and quality appear to the real user of the application. Secondary focus on application availability may be accomplished by synthetic transactions simulating the end user.
  2. Application topology discovery and visualization — The discovery of the software and hardware infrastructure components involved in application execution, and the array of possible paths across which these components communicate to deliver the application.
  3. User-defined transaction profiling — The tracing of user-grouped events, which comprise a transaction as they occur within the application as they interact with components discovered in the second dimension; this is generated in response to a user’s request to the application.
  4. Application component deep dive — The fine-grained monitoring of resources consumed and events occurring within the components discovered in application topology discovery and visualization dimension. This includes server-side components of software being executed.
  5. IT operations analytics — The combination or usage of techniques, including complex operations event processing, statistical pattern discovery and recognition, unstructured text indexing, search and inference, topological analysis, and multidimensional database search and analysis to discover meaningful and actionable patterns in the typically large datasets generated by the first four dimensions of APM. Additionally these data sets are increasingly being analyzed for not only operational information, but business and software analytics.

Inclusion Criteria

 

Vendors will be required to meet the following criteria to be considered for the 2014 APM Magic Quadrant. In comparison to 2013, we have adjusted numerical thresholds.

■ The vendor’s APM product must include all five dimensions of APM (EUM, application topology discovery and visualization, user-defined transaction profiling, application component deep dive, and IT operations analytics). The deep-dive monitoring capabilities must include Java and .NET, but also may include one or more key application component types (e.g., database, application server). The solution must include user-defined transaction profiling, IT operations analytics technologies applied to text and metrics collected by the other four dimensions.

■ The APM product must provide compiled Java and .NET code instrumentation in a production environment.

■ Customer references must be located in at least three of the following geographic locations: North America, South America, EMEA, the Asia/Pacific region and/or Japan.

■ The vendor should have at least 50 customers that use its APM product actively in a production environment.

■ The vendor references must confirm they are monitoring at least 200 production application server instances in a production environment.

■ Some features of the APM offering must be available via a SaaS delivery model. This offering must be delivered directly from the vendor.

■ The product must be shipping to end-user clients for production deployment and designated with general availability by July 15th 2014.

■ Total revenue (including new licenses, updates, maintenance, subscriptions, SaaS, hosting and technical support) must have exceeded $5 million in calendar year 2013.

In addition to these criteria, we will be evaluating the vendor’s ability to cross multiple buying centers, as well as its ability to target specific verticals as validated by reference customers. Detailed criteria and subcriteria will be published along with the final research later this year.

Evaluation Criteria

 

While a vendor may meet the inclusion criteria for the APM Magic Quadrant, placement within the finalized Magic Quadrant will depend on its scoring in a number of categories. Ratings in these categories will be used to determine final placement within the 2014 APM Magic Quadrant. The 2014 evaluation criteria are based on Completeness of Vision and Ability to Execute.

Completeness of Vision

Market Understanding: This criterion evaluates vendor capabilities against future market requirements. The market requirements map to the market overview discussion and look for the following functionality:

■ EUM, including real and synthetic availability testing

■ Runtime application architecture discovery

■ User-defined transaction profiling

■ Application component deep dive

■ IT operations analytics for problem isolation and resolution

■ IT operations analytics to answer questions about software or business execution■ Ability to address the mobile APM market

Marketing Strategy: We evaluate the vendor’s capability to deliver a clear and differentiated message that maps to current and future market demands, and, most importantly, the vendor’s commitment to the APM market through its website, advertising programs, social media, collaborative message boards, tradeshows, training and positioning statements.

Sales Strategy: We evaluate the vendor’s approach to selling APM to multiple buying centers. We also evaluate the vendor’s ability to sell in the appropriate distribution channels, including channel sales, inside sales and outside sales.

Offering (Product) Strategy: We evaluate product scalability, usability, functionality, and delivery model innovation. We also evaluate the innovation related to delivery of product and services.

Business Model: This is our evaluation of whether the vendor continuously manages a well balanced business case that demonstrates appropriate funding and alignment of staffing resources to succeed in this market. Delivery methods will also be evaluated as business model decisions, including the strength and coherence of on-premises and SaaS solutions.

Vertical/Industry Strategy: We evaluate the targeted approaches in marketing and selling into specific vertical industries. Commonly, APM solutions are bought and targeted toward the financial services, healthcare, retail, manufacturing, media, education, government and technology verticals.

Innovation: This criterion includes product leadership and the ability to deliver APM features and functions that distinguish the vendor from its competitors. These include unique approaches to application instrumentation, mobile visibility, and catering towards the increased demands of continuous release. Specific considerations include resources available for R&D, and the innovation process.

Geographic Strategy: This is our evaluation of the vendor’s ability to meet the sales and support requirements of IT organizations worldwide. In this way, we assess the vendor’s strategy to penetrate emerging markets.

Ability to Execute

Product/Service: Gartner evaluates the capabilities, quality, usability, integration and feature set of the solution, including the following functions:

■ Day-to-day maintenance of the product

■ Ease and management of deploying new APM

■ Ease of use and richness of functions within the product

■ Product deployment options and usability

■ Integration of overall APM-related portfolio or unified APM offering

Overall Viability (Business Unit, Financial, Strategy and Organization): We consider the vendor’s company size, market share and financial performance (such as revenue growth and profitability). The leadership within the company in terms of people, what employees think of the leadership, and the ability to drive the company forward. We also investigate any investments and ownership, and any other data related to the health of the corporate entity. Our analysis reflects the vendor’s capability to ensure the continued vitality of its APM offering.

Sales Execution/Pricing: We evaluate the vendor’s capability to provide global sales support that aligns with its marketing messages; its market presence in terms of installed base, new customers, and partnerships; and flexibility and pricing within licensing model options, including packaging that is specific to solution portability.

Market Responsiveness and Track Record: We evaluate the execution in delivering and upgrading products consistently, in a timely fashion, and meeting road map timelines. We also evaluate the vendor’s agility in terms of meeting new market demands, and how well the vendor receives customer feedback and quickly builds it into the product.

Marketing Execution: This is a measure of brand and mind share through client, reference and channel partner feedback. We evaluate the degree to which customers and partners have positive identification with the product, and whether the vendor has credibility in this market.

Customer Experience: We evaluate the vendor’s reputation in the market, based on customers’ feedback regarding their experiences working with the vendor, whether they were glad they chose the vendor’s product and whether they planned to continue working with the vendor. Additionally, we look at the various ways in which the vendor can be engaged, including social media, message boards and other support avenues.

10 Comments »

Category: APM IT Operations Monitoring     Tags:

10 responses so far ↓

  • 1 Criteria for the 2014 Magic Quadrant for Application Performance Monitoring | Euler Global Consulting   May 29, 2014 at 3:01 pm

    [...] Source: Criteria for the 2014 Magic Quadrant for Application Performance Monitoring [...]

  • 2 Simon Levin   May 29, 2014 at 5:13 pm

    Jonah,

    Thanks for publishing these details on the blog. This is the way this information should always be shared to ensure full participation by Gartner clients and non-clients alike. Kudos to you.

    Simon

  • 3 Jonah Kowall   May 29, 2014 at 5:39 pm

    The document has been completed for nearly 3 weeks. There are internal issues, where we’ve had complaints where if criteria are published they are client only, and of course we don’t like to give away content on the blogs, hence there was an impasse. In the past I would share basics and tell people to contact me to get the full document (which I would openly share).

    The result… I’ve just posted it, minus some of the insight, so it would be out in a reasonable amount of time. I’m not waiting for those needed to sign off on the plan, and gone ahead and done the right thing.

    Thanks Simon.

  • 4 John   May 30, 2014 at 10:07 am

    Jonah,

    Clarification-

    “must provide compiled Java and .NET code instrumentation “.

    In last year’s Magic Quadrant, it was stated with an “or”:
    “The APM product must provide compiled Java or .NET code instrumentation in a production environment.”

    It appears support for both Java AND .NET is a must have for inclusion in the 2014 MQ. If a vendor does not have either of these, would it disqualify the vendor?

    -John

  • 5 Jonah Kowall   May 30, 2014 at 8:24 pm

    John you are correct, that is one of the changes this year. We were very close to changing the SaaS requirements as well, but decided to avoid disqualifying too many providers we would delay this one year. With the changes in buying patterns we see full featured SaaS delivery as critical.

  • 6 Rafi Katanasho   June 1, 2014 at 7:33 am

    Good to see the added emphasis on Mobile APM and tightened criteria of complied Java AND .NET.
    Also nice to see analytics broken down into two distinct use cases of diagnostics & business.

    Setting the criteria objectively is perhaps the most challenging task for any analyst as it must strike the right balance between:
    • Objectively addressing the market needs and covering all aspects related to the people, processes, and technology needs for APM;
    • Ensuring that the criteria are not too broad to be all-encompassing as this can potentially lead to misleading comparisons; as well as
    • Ensuring that the criteria are not too rigid or risk the research being limited to a small subset of the wide ranging needs for APM.

    Overall the new criteria are more reflective of the current state of the market needs and drivers; however they don’t adequately address the lifecycle & DevOps aspect of APM.

    Other points to consider (especially in light of the Gartner definition for an application):
    • Provide specific criteria for none Java & .NET apps (as these still make a good percentage of enterprise applications)
    • Given the importance of EUM, Gartner should consider raising the bar higher on this dimension to ensure that there is enough breadth in the offering to address a broader range of application portfolio.
    • With regards to Mobile APM, it would be wise to apply the various APM dimensions to this and mandate crash analytics for component deep-dive.

    Please note that these are my personal analysis @rkatanasho.

  • 7 Jonah Kowall   June 1, 2014 at 8:04 am

    Rafi if you read the mobile APM research you will see that’s the case the same way we expect APM tools to capture logs and stack traces when there are associated issues. (or other diagnostic data).

    EUM has always been weighted heavily, we don’t publish the weighting or sub criteria we use, but you can be assured that’s the case. That will be more easily exposed with the new critical capabilities document we will publish this year as a companion document to the Magic Quadrant.

  • 8 Bibi Zainab   July 10, 2014 at 6:30 am

    In broader way this is right criteriae for APM evaluation. Looking forward for more APM features inclusion.

  • 9 Jonah Kowall   July 10, 2014 at 7:26 pm

    I think you’ll find the critical capabilities helpful, this being the first year of that research it will probably be a work in progress as we tune it in the coming years.

  • 10 APM Framework – End User Experience | CrispyTech.com   November 8, 2014 at 4:14 am

    […] goal of application performance management framework is to provide the end-users with the absolute best experience from beginning-to-end. While most APM […]

Leave a Comment