Will Cappell

A member of the Gartner Blog Network

Will Cappell
Research Vice President
9 years at Gartner
29 years IT industry

Will Cappelli is a Gartner Research VP in the Enterprise Management area, focusing on automation, event correlation and fault analysis, management system architectures, and real-time infrastructure issues. ...Read Full Bio

Coverage Areas:

AI and IAM: Will Two-Tier Analytics Become the Norm for IAM?

by wcappell  |  March 28, 2012  |  10 Comments

Immanuel Kant, in his late 18th century classic, the Critique of Pure Reason, concluded that human cognition takes place on two levels. Level one consists of an initial organization of raw sense materials into geometric figures located in space and time which is then followed, on level two, by a determination of just how these objects and their changes in time relate to one another causally. On the side, Kant also argued that the mental capabilities used to generate results at both levels were the very same capabilities used by human beings to formulate propositions and make inferences about the world.

Various 20th century philosophers have echoed  Kant’s Age of Enlightenment reflections, most notably Wilfred Sellars who railed against the ‘Myth of the Given’, i.e., the idea that raw sense materials lacking any kind of analytical pre-structuring are considered and reasoned about directly. Even given Kant’s influence on modernity, however, it is still interesting to note that many of today’s cognitive scientists and AI practitioners – particularly those working in the field of machine vision – deploy two-tiered models that depend, first, upon mapping of a pixel array into a shape filled three-dimensional space which then must be followed by algorithmic attempts to determine just what kinds of objects those shapes represent.

Although we don’t have the space here to work through all of the arguments, I am inclined to think that Kant and the cognitive scientists have hit on something which is not just true of the processes that govern human cognition but rather reflects the deep structure of any process that seeks to turn volumes of raw, noisy data into information capable of grounding action taken by human beings or machines. Not only does this include most operational technology sense and response systems, it also includes, and perhaps in an exemplary way, the analytics-enhanced performance and event monitoring technology increasingly used by Infrastructure and Application Management (IAM) professionals.

Now, such Kantian processes would be two-tiered. At the first tier, data streams (or stores but the IT wide imperative to reduce latencies of all sorts means streams will increasingly become the paradigm here) would be consumed and transformed into information about objects and their basic relationships to one another in space and time; then, at the second tier, the information is further analyzed with the tasks of sorting objects into types and type hierarchies and establishing causal pathways among the objects becoming paramount.

Market Implications

In terms of analytics-enhanced monitoring system architectures, this means enterprises will, in the long run, deploy functionality on two layers: first, a Complex Event Processing (CEP) or Stream Database (SDB) platform to read and convert packets and other data flows into information about the spatio-temporal state of the systems being monitored, maybe along with some basic space-time data grounded statistical correlations, and, second, an array of causal and type pattern discovery engines that work on the information generated by the first layer. One sign that this is a correct prognostication will be the success of alliances between performance monitoring vendors delivering CEP or SDB functionality (e.g., ExtraHop, Nastel, Optier) with vendors that focus on discovering causal patterns in existing data sets (e.g., Prelert, Verdande, Netuititve.)

It may be worth noting, before I close today, that if it is indeed the case that two tiers are essential to the success of an analytics-enhanced monitoring process, we have a good explanation for why neural networks failed as a technology in the IAM market, despite early signs of promise. At the end of the day, neural network algorithms did organize data into patterns but the patterns were essentially single layered and, as a result, were insufficiently textured or modular to be used by either human or machine. But more on this topic some other time.

10 Comments »

Category: Uncategorized     Tags: , , ,

10 responses so far ↓

  • 1 Charley Rich   March 28, 2012 at 8:52 pm

    First off, I have to admire any blog post that manages to get Immanuel Kant into the text. Wow.

    But seriously, this is a very interesting topic especially with all the ongoing hoopla in the press surrounding “Big Data”. However, most of the articles in the media focus on the performance issues related to the “big” word and the new technologies that are required to store and retrieve it. But the author of this post takes it a step further. We also have to find a way to make sense of the data and act on it. Think decision support…

    I think we can take the author’s analogy of how the brain works to a two-tiered process and also add into the model how organizations work. We all moan about how IT and the business always have a divide between them and never fully bridge that gap. Perhaps, our organizational structures also require a two tiered model just like physical vision and object recognition do. In that case the first tier using a low latency technology like CEP can deliver great value to IT in the problem management space enabling them to be more proactive in problem resolution. While the second tier, self-learning and able to understand causality can be very useful for determining business impact something that is always changing.

  • 2 wcappell   March 29, 2012 at 11:01 am

    Thanks, Charley. The second paragraph of your comment contains a critical point. The focus of much Big Data discussion, at least in the IAM realm, has been on how NoSQL technologies make really big data stores possible not on the analytics that will make those really bid data stores useful. There are two issues here:

    First, the NoSQL technologies can do what they do largely by stripping out semantics. That makes analytics technology all that more important because it needs to pump the semantics back into the data before the true analysis can even get under way.

    Second, many of the analytics algorithms will require the ability to analyze data from a global perspective and that could undermine the parallelism enabled by MapReduce and its cousins. Paying attention to the ‘big’ word without keeping the need to analyze in mind will lead to disappointments down the road.

  • 3 William Louth   March 29, 2012 at 1:43 pm

    You seem to think that we (humans or operators or orgs) are the main consumers of the data when in fact it should be the software itself. If we want to design and deploy truly self aware, self learning and self adaptive software then the data must be kept largely local and acted on in the context of the process and task itself.

    http://www.jinspired.com/site/feedback-control-signals-its-arrival-in-the-cloud-enterprise

    We don’t need data stores (or streams) especially big ones. The mere mentioning of \tiers\ or \layers\ is going to have all those involved creating boundaries and borders that do not need to exist both in terms of space (storage) and time.

    http://www.jinspired.com/site/the-complexity-and-challenges-of-it-management-within-the-cloud

    CEP has largely failed to gain the traction is so rightly deserves because its not embedded in the software and used to generate signals and implement feedback loops that influence the execution behavior of the software (the same software which reflects more of the business execution than ever before).

    \we need for IT to change starting with how it (or its systems) observe. Moving from logging to signaling. Moving from monitoring to metering. Moving from correlation to causation. Moving from process to code then context. Moving from state to behavior then traits. Moving from delayed to immediate. Moving from past to present. Moving from central to local. Moving from collecting to sensing. When that has occurred we can then begin to control via built in controllers and supervisors.\

    http://www.jinspired.com/site/the-complexity-and-challenges-of-it-management-within-the-cloud

  • 4 wcappell   March 29, 2012 at 4:41 pm

    Thank you for your comment, William. You raise a number of important points that I’d like to address.

    First, you are absolutely right that we should not confine our considerations to the human consumption of analyzed performance and event data. In fact, that was precisely the point I was trying to make above in stating that the two-tier model should not be seen only as description of human cognitive processes but rather as design prescription for any sense-response system. Having said that, I would not minimize the importance of the case of human consumption.

    Second, human cognition does indeed seem to be modular at some level, even if those modules are porous and can sometimes take over roles from one another. In this post, I am speculating that a reasonably well established modularization takes place within human cognitive capacity (the division of the construction of spatio-temporal models from the construction of causal models.)This modularization is not just an accident of evolutionary biology. Rather, it reflects something essential to any system (human being, machine, group of humans, groups of machines, aliens, etc.) that seeks to capture and analyze data in a way that makes the product of the analysis fit to serve as input into further action.

    Third, and finally, I completely agree that one of the reasons for CEP’s comparatively lukewarm reception by the market to date is a result of the isolation of the CEP engines from a broader analytic environments, i.e., their embedding into just the kind of two-tier systems advocated in this post.

  • 5 William Louth   March 30, 2012 at 12:02 pm

    The reason that I diminish our involvement in the process of “monitoring” is that we are terrible as operators and incredibly slow in terms of response time (relatively to the dynamic & temporal aspects of the execution speeds and workload variations). We are much better learning from observed behavioral patterns creating suitable models for reasoning, education and verifying but slow in detecting/sensing such patterns as they happen at scale and acting in time. The future management/monitoring model such not be one based on measurement of application behavior and resource consumption but based on signals generated & actions taken from & by control (cep) agents embedded within runtimes…we effectively manage by proxy…the new model for ops consists of signals and (corrective/protective) actions…we manage the feedback loops within such systems and not directly the applications and requests.

    This is somewhat like how QoS can be implemented in Applications:

    http://www.jinspired.com/wp-content/uploads/2011/11/QoSIntro.pdf

  • 6 Stephen Dodson   March 30, 2012 at 2:31 pm

    Interesting post, and the two phases of analysis align with my experience working with this type of data and what we see in the market.

    In my experience of applying causal pattern recognition and machine learning on data, there are two steps:

    1. Pre-process the data to give it structure (encoding the knowledge into a model)
    2. Use machine learning to identify causal patterns in the structured data based on probability (reasoning)

    A simple example of this is log file analysis. Timestamps in raw unstructured log data need to be identified or set to give the information temporal context prior to pattern recognition.

    In terms of vendor tools, the division between the tasks of structuring data and machine-learning is blurred. For example, tools such as Splunk or CA Introscope APM will give the data temporal context and some attributes. However, pattern recognition requires additional pre-processing to create the model and reduce the dimensionality of the data.

    In the current landscape we see vendors who are collecting and warehousing large volumes of data being data sources for our analysis, and others, like ourselves at Prelert, focusing on the 2nd tier of analysis.

  • 7 Verdande Technology Blog « Verdande Technology Blog   April 3, 2012 at 8:53 pm

    [...] Will Cappell, Gartner Research VP, recently published an article referencing Verdande Technology’s unique software platform that can predict future events in advance by “discovering casual patterns in existing data sets.” Read the full article>> [...]

  • 8 Making Big Data into Small Data – SYS - Learn all about Performance Monitoring - Performance Monitoring   April 9, 2012 at 4:14 pm

    [...] post but last week Gartner Analyst Will Cappelli did just that. As you’ll see in his post, “AI and IAM: Will Two-Tier Analytics Become the Norm for IAM?” context is the key. Cappelli addresses Kant’s conclusion that human reasoning is a two-tier [...]

  • 9 david ricardo   April 12, 2012 at 9:59 pm

    Hi there I am so thrilled I found your web site, I really found you by accident, while I was browsing on Aol for something else, Anyhow I am here now and would just like to say thank you for a marvelous post and a all round enjoyable blog (I also love the theme/design), I don’t have time to read it all at the moment but I have book-marked it and also added in your RSS feeds, so when I have time I will be back to read much more, Please do keep up the superb job.

  • 10 uprawy roslin w Danii   April 13, 2012 at 11:06 pm

    I’d always want to be update on new blog posts on this internet site , saved to favorites !