Gartner Blog Network


Inclusion Criteria for the 2018 Network Performance Monitoring and Diagnostics Magic Quadrant

by Sanjit Ganguli  |  July 10, 2017  |  Submit a Comment

The 2018 NPMD Magic Quadrant and Critical Capabilities research process has kicked off, and Vivek Bhalla, Pankaj Prasad and I are excited to be lead authors. The inclusion criteria for both research notes are listed below. If your organization meets the criteria, please reach out to MQ.NetworkPerfMonitoringDiagnostics@gartner.com before July 14th, 2017.

 

  1. Magic Quadrant Market Definition

Gartner’s view of the market is focused on transformational technologies or approaches delivering on the future needs of end users. It is not focused on the market as it is today.

Gartner defines the Network Performance Monitoring and Diagnostics Market (herein referred to as NPMD) as:

NPMD tools allow for network engineers to understand the performance of applications and infrastructure components via network instrumentation. Additionally, these tools provide insight into the quality of the end user’s experience. The goal of NPMD products is not only to monitor the network components to facilitate outage and degradation resolution, but also to identify performance optimization opportunities. This is conducted via diagnostics, analytics and debugging capabilities to complement additional monitoring of today’s complex IT environments.

This market is a fast-growing segment of the larger network management space. Gartner estimates the size of the NPMD tools market at $2.1 billion (https://www.gartner.com/document/3732135).

Note that analysts may need to update the market definition as they progress through the research process.  You will be explicitly notified of the change(s) should they need to happen.

 

  1. Magic Quadrant Inclusion Criteria

The inclusion criteria represent the specific attributes that analysts believe are necessary for inclusion in this research.

To qualify for inclusion, vendors will be required to meet the following criteria to be considered for the 2018 NPMD Magic Quadrant and Critical Capabilities:

  • The ability to monitor, diagnose and generate alerts for:
    • Network endpoints — Servers, virtual machines, storage systems or anything with an IP address by measuring these components directly in combination with a network perspective. This includes IoT, cloud-hosted and wireless endpoints.
    • Network components — Such as routers, switches and other network devices. This includes SDN and NFV components.
    • Network links — Connectivity between network-attached infrastructure.
  • The ability to monitor, diagnose and generate alerts for dynamic end-to-end network service delivery as it relates to:
    • Digital experience monitoring — The capture of data about how end-to-end application availability, latency and quality appear to the end user from a network perspective. This is limited to the network traffic visibility and not within components such as what application performance monitoring (APM) is able to accomplish.
    • Business transaction monitoring — The speed and overall quality of network service and/or application delivery to the user in support of key business activities, as defined by the operator of the NPMD product. These definitions may overlap as services and applications are recombined into new applications.
    • Infrastructure component interactions — The focus on infrastructure components as they interact via the network, as well as the network delivery of services or applications. This includes the ability to provide network path analytics.
  • Support for analysis of:
    • Real-time performance and behaviors — Essential for troubleshooting in the current state of the environment. Analysis of data must be done within three minutes under normal network loads and conditions.
    • Historical performance and behaviors — To help understand what occurred or what is trending over time.
    • Predictive behaviors by leveraging analytics technologies — The ability to distill and create actionable advice from the large dataset collected across the various data sources (include reference to note).
  • Leverage the following data sources:
    • Network device generated traffic data, including flow-based data sources inclusive of NetFlow and IPFIX.
    • Network device generated health data.
    • Raw network packet analysis to identify application types and performance characteristics.
  • The ability to support the following scalability and performance requirements:
  • Real-time monitoring of 10 gigabit (10G) Ethernet networks at full line rate.
  • Ingest sampled flow records at a rate of 75,000 flows per second via a single instance of the product.

Non-product Related Criteria

  • Total NPMD product revenue (including new licenses, updates, maintenance, subscriptions, SaaS, hosting and technical support) must have exceeded $7.5 million for Q3 CY2016 through Q2 CY2017, excluding revenue derived from security-related buying centers.
  • The vendor must have at least 75 customers that use its NPMD product actively in a production environment.
  • The vendor must have at least 10 customers located in at least two of the following geographic locations: North America, Latin America, EMEA and/or APAC that use its NPMD product actively in a production environment.

Please take time to analyze the inclusion criteria and respond to this notification on your ability to qualify for inclusion or not.  Indicate the name(s) of the product(s)/service(s) your organization offers in this market. 

Note that analysts may need to update the inclusion criteria as they progress through the research process.  You will be explicitly notified of the change(s) should they need to happen.

 

 

  1. Magic Quadrant Evaluation Criteria and Weights

Note that analysts may need to update the evaluation criteria and associated weights as they progress through the research process.  You will be explicitly notified of the change(s) should they need to happen.

 

Completeness of Vision

Gartner analysts evaluate vendors on their ability to convincingly articulate logical statements.  This includes current and future market direction, innovation, customer needs, and competitive forces and how well they map to Gartner’s view of the market

Evaluation Criteria Definition 2018 Weight 2017

Weight

Market Understanding Ability to understand customer needs and translate them into products and services. Vendors that show a clear vision of their market – listen, understand customer demands, and can shape or enhance market changes with their added vision.

 

This criterion evaluates vendor capabilities against future market requirements. The market requirements map to the market overview discussion and look for the following functionality:

 

·         Data source support, including application visibility

·         Analytics

·         Diagnostic workflows

·         Real-time monitoring

·         Virtualization (NFV and SDN)

·         Hybrid IT environments

 

High High
Marketing Strategy Clear, differentiated messaging consistently communicated internally, externalized through social media, advertising, customer programs, and positioning statements.

 

We evaluate the vendor’s capability to deliver a clear and differentiated message that maps to current and future market demands, and, most importantly, the vendor’s marketing effectiveness to the NPMD market through its website, advertising programs, social media, collaborative message boards, tradeshows, training and positioning statements.

Standard Standard
Sales Strategy A sound strategy for selling that uses the appropriate networks including:  direct and indirect sales, marketing, service, and communication.  Partners that extend the scope and depth of market reach, expertise, technologies, services and their customer base.

 

We evaluate the vendor’s approach to selling NPMD in the appropriate distribution channels, including channel sales, inside sales and outside sales.

Standard Standard
Offering (Product) Strategy An approach to product development and delivery that emphasizes market differentiation, functionality, methodology, and features as they map to current and future requirements.

 

We evaluate product scalability, usability, functionality, and delivery model innovation. We also evaluate the innovation related to the delivery of products and services

High High
Business Model The design, logic and execution of the organization’s business proposition to achieve continued success.

 

This is our evaluation of whether the vendor continuously manages a well-balanced business case that demonstrates appropriate funding and alignment of staffing resources to succeed in this market. Delivery methods will also be evaluated as business model decisions, including the strength and coherence of on-premises and SaaS solutions.

Low Low
Vertical/Industry Strategy The strategy to direct resources (sales, product, development), skills, and products to meet the specific needs of individual market segments, including verticals.

 

We evaluate the targeted approaches in marketing and selling into specific vertical industries.

Low Low
Innovation Direct, related, complementary, and synergistic layouts of resources, expertise or capital for investment, consolidation, defensive or pre-emptive purposes.

 

This criterion includes product leadership and the ability to deliver NPMD features and functions that distinguish the vendor from its competitors. Specific considerations include resources available for R&D and the innovation process.

High High
Geographic Strategy The vendor’s strategy to direct resources, skills and offerings to meet the specific needs of geographies outside the “home” or native geography, either directly or through partners, channels and subsidiaries, as appropriate for that geography and market.

 

This is our evaluation of the vendor’s ability to meet the sales and support requirements of IT organizations worldwide. In this way, we assess the vendor’s strategy to penetrate emerging markets.

Low Low

 

 

 

Ability to Execute

Gartner analysts evaluate vendors on the quality and efficacy of the processes, systems, methods or procedures that enable IT provider performance to be competitive, efficient and effective, and to positively impact revenue, retention and reputation within Gartner’s view of the market.

Evaluation Criteria Definition 2018 Weight 2017

Weight

Product/Service Core goods and services that compete in and /or serve the defined market. This includes current product and service capabilities, quality, feature sets, skills etc.  This can be offered natively or through OEM agreements/partnerships as defined in the market definition and detailed in the sub-criteria.

 

Gartner makes judgments from a variety of inputs in order to evaluate the capabilities, quality, usability, integration and feature set of the solution, including the following functions:

 

·         Data source support, including application visibility

·         Analytics

·         Diagnostic workflows

·         Real-time monitoring

·         Day-to-day maintenance and management of the product

·         Ease and simplicity of deployment and configuration

·         Ease of use and richness of functions within the product

·         Product deployment options and usability

 

High High
Overall Viability (Business Unit, Financial, Strategy, Organization): Financials Viability includes an assessment of the organization’s overall financial health as well as the financial and practical success of the business unit.  Views the likelihood of the organization to continue to offer and invest in the product as well as the product position in the current portfolio.

 

We consider the vendor’s company size, market share and financial performance (such as revenue growth and profitability). We also investigate any investments and ownership, and any other data related to the health of the corporate entity. Our analysis reflects the vendor’s capability to ensure the continued vitality of its NPMD offering.

High High
Sales Execution/Pricing The organization’s capabilities in all pre-sales activities and the structure that supports them. This includes deal management, pricing and negotiation, pre-sales support and the overall effectiveness of the sales channel.

 

We evaluate the vendor’s capability to provide global sales support that aligns with its marketing messages; its market presence in terms of installed base, new customers and partnerships; and flexibility and pricing within licensing model options, including packaging.

Standard Standard
Market Responsiveness and Track Record Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve, and market dynamics change. This criterion also considers the vendor’s history of responsiveness to changing market demands.

 

We evaluate the execution in delivering and upgrading products consistently, in a timely fashion, and meeting road map timelines. We also evaluate the vendor’s agility in terms of meeting new market demands, and how well the vendor receives customer feedback and quickly builds it into the product.

High High
Marketing Execution The clarity, quality, creativity and efficacy of programs designed to deliver the organization’s message in order to influence the market, promote the brand, increase awareness of products and establish a positive identification in the minds of customers. This “mind share” can be driven by a combination of publicity, promotional, thought leadership, social media, referrals and sales activities.

 

 

This is a measure of brand and mindshare through client, reference and channel partner feedback. We evaluate the degree to which customers and partners have positive identification with the product, and whether the vendor has credibility in this market.

Standard Standard
Customer Experience Products and services and/or programs that enable customers to achieve anticipated results with the products evaluated. Specifically, this includes quality supplier/buyer interactions technical support, or account support. This may also include ancillary tools, customer support programs, availability of user groups, service-level agreements, etc.

 

We evaluate the vendor’s reputation in the market, based on customers’ feedback regarding their experiences working with the vendor, whether they were glad they chose the vendor’s product and whether they planned to continue working with the vendor. Additionally, we look at the various ways in which the vendor can be engaged, including social media, message boards and other support avenues.

High High
Operations The ability of the organization to meet goals and commitments. Factors include: quality of the organizational structure, skills, experiences, programs, systems and other vehicles that enable the organization to operate effectively and efficiently.

 

The ability of the organization to meet goals and commitments. Factors include: quality of the organizational structure, skills, experiences, programs, systems and other vehicles that enable the organization to operate effectively and efficiently.

Not Rated Not Rated

 

  1. Critical Capabilities Inclusion Criteria

The inclusion criteria for the Critical Capabilities research is the same as the Magic Quadrant for NPMD. Your survey submission and product demo will also be used for the purposes of writing this document in addition to the Magic Quadrant.  The inclusion criteria for the Critical Capabilities research is the same as the Magic Quadrant for NPMD (see part 5 of this document)

 

  1. Critical Capabilities Use Case Definition and Critical Capabilities Definition

The 2018 NPMD Critical Capabilities will be published as a complementary piece of research to the 2018 NPMD Magic Quadrant

The 2018 NPMD Critical Capabilities will be assessed upon the following criteria:

  • End-point/Component/Link Monitoring
  • Business Transaction Monitoring
  • Diagnostics
  • Analytics
  • Integration and Interoperability

Each of these Critical Capabilities is defined as the following:

  • Endpoint/Component/Link Monitoring

The ability to monitor, diagnose and generate alerts for endpoints, e.g., servers, virtual machines (VMs), storage systems or anything with an IP address, including IoT, cloud-hosted and wireless endpoints; network components (e.g., routers, switches and other network devices); and network links (connectivity between network-attached infrastructure).

  • Business Transaction Monitoring

Business transaction dashboards are key to all use cases, because they help IT organizations understand alignment with IT objectives, as shown by key performance indicators (KPIs). Reporting is evaluated on the product’s ability to provide trend reports that measure productivity across core processes.

Dashboard and reporting capabilities must be tied to business value metrics, common critical success factors and KPIs. The tool must offer multidimensional charts that show how related metrics affect each other. These would be based on the tools’ ability to monitor, diagnose and generate alerts for dynamic end-to-end network service delivery as it relates to the end-user experience, business service delivery and infrastructure component interaction.

  • Diagnostics

The ability to facilitate or automate typical steps taken by an operator to troubleshoot, triage and diagnose issues. Aspects that enhance a tool’s diagnostics include contextual drill-down, intuitive workflow, navigational aids (such as hover-over features) and simple search mechanisms.

  • Analytics

The ability to leverage one or more algorithmic techniques to distill, infer and/or create action-oriented advice from the dataset collected, which can include metric, time-series based and unstructured data sources.

Algorithmic capabilities include the ability to support model discovery, ascertain the root causes of performance problems, assess the impacts of multiple known root causes, predict events and recommend remediation; model discovery is the basis of the other four use cases. Many NPMD tools claiming algorithmic capabilities often only filter, aggregate and visualize data. Such tools do not perform inferences; at best, they only facilitate inferences taking place within a human mind (diagnostics). It is important to understand the distinction, whether automated model discovery, root cause analysis, cause impact determination, prediction or action recommendation is the desired requirement.

  • Integration and Interoperability

The tools’ ability to be integrated with other third-party tools and the ability of those third-party tools to integrate with NPMD tools are increasingly important, particularly for organizations that license best-of-breed ITOM products, as opposed to products from a single vendor.

Products are evaluated on the presence of out-of-the box integration capabilities (via APIs or connectors) that link to network fault monitoring (NFM), network automation, event correlation and analytics (ECA) tools, application performance monitoring (APM), IT infrastructure monitoring (ITIM), Artificial Intelligence for IT Operations (AIOps) platforms and unified communications monitoring (UCM) tools.

The Use Cases are defined as the following:

  • Network Operator

Responsibilities include troubleshooting network-related hardware/software, monitoring and configuring network devices, and making recommendations to network architects.

Network operators are typically members of a team that is responsible for monitoring and maintaining the network infrastructure. Extended responsibilities may also include the analysis of network components for correct configuration (including patching), upgrades and making recommendations for network enhancements. In enterprises and government organizations, they may also serve as liaisons with communications service providers to ensure the proper service delivery, operation and compliance of the provider’s equipment, as well as to ascertain and delineate responsibility when fault isolation is not immediately apparent.

  • Network Architect

Responsibilities include conceptual network design, planning, governance, communicating architecture rationale to technical and business stakeholders, and escalation handling.

Unlike generalists, engineers and operators, architects are largely proactive and work predominantly on planning, documentation and optimization, new projects and infrastructure strategy. Network architects often report outside the infrastructure and operations teams, typically working in an enterprise architecture or project management office. Network architects are relied on to provide functional, technical and/or process leadership. The formulation of best practices to ensure effective workflow and the monitoring of industry/technological trends, adoption and direction are common extended responsibilities.

  • IT Operations Generalist

Responsibilities include end-user support, fault validation, service desk ticket generation, troubleshooting broad IT issues and working with multiple domain specialists.

IT operations generalists are typically members of a team that is responsible for the monitoring and support of the IT Infrastructure. This can include endpoints, servers, applications and/or the network (including voice and video services). Given their broad remit, IT operations generalists are normally interfacing with fault and availability monitoring tools and, in less mature organization, they deal with issues reactively, as they are being reported by end users. Collaboration with server, database, application support, application development, network, storage, and security teams is frequently the case. As a result, fault identification and isolation is a frequent task, as is problem handoff.

  • IT Operations Manager

Responsibilities include team management, service compliance, escalation handling, cross-domain knowledge and experience, report generation and continuous service improvement.

IT operations managers typically oversee a team of IT operations generalists. As a result, they are often a point of escalation and require adequate high-level visibility as to the immediate health of business applications and services, as well as the underlying infrastructure that supports them. Historical and performance reporting assists IT operations managers in moving to more-proactive states, whereby emerging issues can be anticipated and dealt with prior to end users being affected. Tracking and reporting of SLA compliance, as well as identifying cross-domain constraints, are also requirements that an IT operations manager would seek from any tool.

  • Line-of-Business User

Responsibilities include planning and reporting divisional and regional activities and performance in the service of business objectives (e.g., growth, cost and risk).

The LOB user may be a manager or member of a team that is either wholly or partially responsible for those applications or services that are perceived to be vital to running the enterprise or business unit.  A LOB user will have a requirement to collect, identify and analyze business data to review and evaluate the organization’s business performance. Identifying constraints to the delivery of the service and impediments to maximizing its efficiency and performance will be priorities. Adherence to regulatory compliance, while reporting on KPI trends and patterns, is also vital.

Category: 

Sanjit Ganguli
Research Director
2 years at Gartner
17 years IT Industry

Sanjit Ganguli is a Research Director with Gartner's IT Systems, Security and Risk research group. His primary focus is on network and application performance management. Read Full Bio




Leave a Reply

Your email address will not be published. Required fields are marked *

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.