Jonah Kowall

A member of the Gartner Blog Network

Jonah Kowall
Research Vice President
2 years with Gartner
18 years IT industry

Jonah Kowall is a research Vice President in Gartner's IT Operations Research group. He focuses on application performance monitoring (APM), event correlation and analysis (ECA), network management systems (NMS), network performance management (NPM), network configuration and change management (NCCM), and general system and infrastructure monitoring technologies. Read Full Bio

Coverage Areas:

Web-Scale Monitoring and the Importance of APM

by Jonah Kowall  |  December 27, 2013  |  4 Comments

Cameron Haight (@cameron_haight‎) and myself recently published research on how monitoring is applied to web-scale environments. Companies such as Amazon, Google, and Facebook run their environments using different fundamentals than typical enterprise IT organizations. This includes changes in infrastructure, management software, and the applications running on the infrastructure (among many other things including people and process which we don’t get into in this research).

In this research we cover some of the core fundamentals of both open source and commercial software systems which can support and often times are built with the same fundamental differences that distinguish web-scale environments. Many of these elements have to do with eventual consistency, size/scale, volatility, and the required performance of the applications which customers/consumers demand.

Further in the research we investigate the different ways data is collected, and once collected the elements of visualization, and analytics done by the user and the software to bring forth meaning in the vast amount of data collected.

We were able to build a presentation at the recent Gartner Data Center Conference in early December (in Las Vegas) where we converted this content and material into a presentation which looked at similar topics. We did a bunch of polling, which I should have results from in the next couple weeks. In the presentation we also dug into some of the open source (statsd, collectd, Graphite, and other associated projects for metric collection) and vendor supplied tools including those from AppDynamics, AppFirst, Boundary, Circonus, Data Dog, Librato, New Relic, Sumo Logic, and Splunk.

You can find the research here (sorry clients only) : http://www.gartner.com/document/2633831

4 Comments »

Category: Analytics APM Big Data DevOps IT Operations Logfile Monitoring SaaS Trade Show     Tags:

4 responses so far ↓

  • 1 Web-Scale Monitoring and the Importance of APM ...   December 28, 2013 at 3:28 pm

    [...]   [...]

  • 2 GP   January 7, 2014 at 5:20 am

    Jonah – I tried to access the report you reference above but it does not have the content you describe in your comments. It is interesting that you have included open source in your analysis. Companies like Netflix and LinkedIn (amongst others) are helping define scalable and robust reference architectures for real-time data ingestion and analysis.

    E.g. Netflix just released SURO.
    http://openopsiq.com/2013/12/15/announcing-suro-backbone-of-netflixs-data-pipeline/

    Also, LinkedIn just released a fairly detailed write-up on their unified logging approach.
    http://openopsiq.com/2013/12/29/the-log-what-every-software-engineer-should-know-about-real-time-datas-unifying-abstraction/

    Technologies like Apache Flume, Apache Kafka (for collection), Real-time processors like Apache Storm, Apache Spark, Apache Samza and end-user analytics tools including ElasticSearch, Kibana, GrayLog2, Graphite and Druid offer a wide variety of powerful and proven solutions.

    A lot of business drivers for enterprise adoption is coming from security and compliance mandates around CDM (Continuous Diagnostic Monitoring) and other OCC/Federal driven mandates for securing critical national assets in the face of cyberthreats.

  • 3 Jonah Kowall   January 7, 2014 at 1:58 pm

    Pretty detailed for sure, but it doesn’t speak about use cases or how logs should be shared and socialized. The link is correct, but this post blended what we presented (if you are a client I’m happy to share the presentation content) and what we published. The detail and audience was different across both documents hence the reason the blog was more broad.

  • 4 GP   January 7, 2014 at 8:09 pm

    Jonah – yes we are a Gartner client :)

Leave a Comment