Gartner Blog Network


More On Big Data Security Analytics Readiness

by Anton Chuvakin  |  September 23, 2013  |  4 Comments

I was reading an excellent GTP piece on big data analytics called “Hadoop and MapReduce: Big Data Analytics” the other day. It is a very useful assessment of big data technologies and approaches. The author is not writing about the security use of Hadoop, but assesses how big data analytics differs from traditional data analysis (OLTP, ODS, OLAP, EDW, BI, CEP alphabet soup) across all domains. The piece would come handy to educate all the screeching big data monkeys – if only they’d be convinced to read a 41 page document….

But I digress. What struck me in this research are these two pictures where he compares the approaches based on data volume and processing complexity:

bd-data-structure

bd-proc-complexity

They have relevance for [attempted] use of big data analytics for security. To understand my point, you only need to know what “CEP” stands for – Complex Event Processing (aka “correlation” in SIEM land). Now look at the right picture: if you think that “SIEM correlation is complex”, you should not even try to come within a 5 mile radius of a Hadoop install since it is MUCH harder. Further, look at the left picture: if you mostly deal with nicely structured data (even if lots of it), big data methods may not be the best.

Keep this in mind the next time somebody tries to sell you a “Hadoop cluster with a Pig on top in place of a SIEM” vision….

Finally, a few insightful quotes from the same document:

  • “The tools/techniques, architectural patterns, and use cases where big data analysis is applicable is very much a work in progress.” (therefore you MUST be prepared to experiment, rather than consume)
  • “Big data analytics is undergoing huge (at times chaotic) change and innovation. Big data analysis is new and has few best practices; expect to learn through failures and the success of peers.” (therefore, what you know to be true today may not be tomorrow)
  • “Installing, configuring, and administering a production-scale Hadoop cluster requires considerable system administration expertise. Interacting with Hadoop requires a detailed knowledge of programming languages.” (if you think SQL is hard, MapReduce will kill ya dead…)
  • “Data sampling is dead, and the success of a big data analytics initiative cannot be shown with a small-scale pilot.” (and, no, Excel is not a big data tool)

Just some food for thought….

Related posts:

Category: analytics  big-data  security  

Anton Chuvakin
Research VP
5+ years with Gartner
16 years IT industry

Anton Chuvakin is a research VP at Gartner's GTP Security and Risk Management group. Before Mr. Chuvakin joined Gartner, his job responsibilities included security product management, evangelist… Read Full Bio


Thoughts on More On Big Data Security Analytics Readiness


  1. William Chesher says:

    Big Data Security Analytics is achievable.

    Many of the lessons from the Business Intelligence world have not yet been applied to security analytics.

    1) There is a deeper understanding of risk in companies that deal in risk as a day in day out commodity.

    2) Organizations of these types are:

    – Big Tobacco
    – Insurance
    – Medical Device Manufacturers
    – Pharma
    – Financials

    3) Companies like these tend to view information security problems as part of a more holistic set of problems and potential solutions in the context of a larger “Information Risk” umbrella.

    The possibility of Big Data security analytics is not only achievable, but the concept of an Enterprise Information Risk Management Platform (EIRMP) is a real possibility with integrated BI to help manage risk using more than just security data.

    Big Data allows us to enmesh PeopleSoft performance review data with travel system data, coupled with corporate spending and badging system data, along with external attack pattern records, and internal network traffic data, combined with local weather patterns and traffic, for example, all together.

    The Information Risk analytics capabilities promised by new data technologies are very promising, indeed.

  2. William, thanks a lot for the insightful comment! Indeed, BI + RDBMS (not big data) is where orgs should probably look first, before they delve into the largely uncharted land of Hadoop et al….

  3. […] More On Big Data Security Analytics Readiness […]



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.