Gartner Blog Network

Web-Scale Monitoring and the Importance of APM

by Jonah Kowall  |  December 27, 2013  |  4 Comments

Cameron Haight (@cameron_haight‎) and myself recently published research on how monitoring is applied to web-scale environments. Companies such as Amazon, Google, and Facebook run their environments using different fundamentals than typical enterprise IT organizations. This includes changes in infrastructure, management software, and the applications running on the infrastructure (among many other things including people and process which we don’t get into in this research).

In this research we cover some of the core fundamentals of both open source and commercial software systems which can support and often times are built with the same fundamental differences that distinguish web-scale environments. Many of these elements have to do with eventual consistency, size/scale, volatility, and the required performance of the applications which customers/consumers demand.

Further in the research we investigate the different ways data is collected, and once collected the elements of visualization, and analytics done by the user and the software to bring forth meaning in the vast amount of data collected.

We were able to build a presentation at the recent Gartner Data Center Conference in early December (in Las Vegas) where we converted this content and material into a presentation which looked at similar topics. We did a bunch of polling, which I should have results from in the next couple weeks. In the presentation we also dug into some of the open source (statsd, collectd, Graphite, and other associated projects for metric collection) and vendor supplied tools including those from AppDynamics, AppFirst, Boundary, Circonus, Data Dog, Librato, New Relic, Sumo Logic, and Splunk.

You can find the research here (sorry clients only) :

Category: analytics  apm  big-data  devops  it-operations  logfile  monitoring  saas  trade-show  

Jonah Kowall
Research Vice President
3.5 years with Gartner
20 years IT industry

Jonah Kowall is a research Vice President in Gartner's IT Operations Research group. He focuses on application performance monitoring (APM), Unified Monitoring, Network Performance Monitoring and Diagnostics (NPMD), Infrastructure Performance Monitoring (IPM), IT Operations Analytics (ITOA), and general application and infrastructure availability and performance monitoring technologies. Read Full Bio

Thoughts on Web-Scale Monitoring and the Importance of APM

  1. GP says:

    Jonah – I tried to access the report you reference above but it does not have the content you describe in your comments. It is interesting that you have included open source in your analysis. Companies like Netflix and LinkedIn (amongst others) are helping define scalable and robust reference architectures for real-time data ingestion and analysis.

    E.g. Netflix just released SURO.

    Also, LinkedIn just released a fairly detailed write-up on their unified logging approach.

    Technologies like Apache Flume, Apache Kafka (for collection), Real-time processors like Apache Storm, Apache Spark, Apache Samza and end-user analytics tools including ElasticSearch, Kibana, GrayLog2, Graphite and Druid offer a wide variety of powerful and proven solutions.

    A lot of business drivers for enterprise adoption is coming from security and compliance mandates around CDM (Continuous Diagnostic Monitoring) and other OCC/Federal driven mandates for securing critical national assets in the face of cyberthreats.

    • Jonah Kowall says:

      Pretty detailed for sure, but it doesn’t speak about use cases or how logs should be shared and socialized. The link is correct, but this post blended what we presented (if you are a client I’m happy to share the presentation content) and what we published. The detail and audience was different across both documents hence the reason the blog was more broad.

  2. GP says:

    Jonah – yes we are a Gartner client :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.