Gartner Blog Network


“Big Analytics” for Security: A Harbinger or An Outlier?

by Anton Chuvakin  |  March 26, 2012  |  6 Comments

You have 10 petabytes of security data in your Hadoop cluster.

You count RAM in terabytes and CPU cores in dozens.

You speak HiveQL better than you speak English.

You collect literally and unquestionably every timed record of activity in your organization – including transaction logs, IM messages, flows, anything.

You run queries over 13 months of data – and you do not have to take a vacation before the results come in.

You outgrew your market-leading SIEM product … 5 years ago.

You have statisticians (data scientists) on speed-dial – and on staff.

You run statistical models on volumes of security data before your morning coffee – and get good results.

Your organizations’ BI team thinks you are actually cool… despite being in security.

So….

are you a HARBINGER or an OUTLIER?

Is this the way information security will be done nearly everywhere in 3, 5, 10 years? (good arguments for this)

Or is this a case of “there are only 10 organizations in a Top 10 list”? (some arguments for this)

Is this the way we all need to learn to succeed with current and future threats?

Or is this the way to the top of the mountain that only the enlightened gurus will ever tread?

In any case, let’s keep this discussion going!

 

P.S.  By the way, remember that:  “If at first you don’t succeed, skydiving may not be for you.”  [by unknown] –> “If you keep failing with small data now, BIG DATA isn‘t for you!” [by Anton Chuvakin]

Category: analytics  big-data  logging  security  siem  

Tags: analytics  big-data  security  security-monitoring  

Anton Chuvakin
Research VP and Distinguished Analyst
5+ years with Gartner
17 years IT industry

Anton Chuvakin is a Research VP and Distinguished Analyst at Gartner's GTP Security and Risk Management group. Before Mr. Chuvakin joined Gartner, his job responsibilities included security product management, evangelist… Read Full Bio


Thoughts on “Big Analytics” for Security: A Harbinger or An Outlier?


  1. […] original post here: “Big Analytics” for Security: A Harbinger or An Outlier? Comments […]

  2. Shaleen Shah says:

    I had a laugh reading this, ingenious! I think I can relate. Isn’t it great if we can predict a disaster from happening thousands of miles away? I think, I’ll be an outlier for now and observe the buzz around Big Data et al.

  3. Probably a harbinger, based especially on the leading work done by Zions Bank using this technology stack, people competencies and analytic methods.

    Wider adoption will come from maturation of the technology stack that will reduce the need for such specialized skills and level of effort. But the basic appeal of Big Data for SIEM/log management security remains: huge data volumes, ability to hold disparate data sets of varied formats, relatively cheap processing, etc.

    ironically, some of the first generation of SIEMs and log management products were built on proprietary file management systems (e.g. Network Intelligence/RSA enVision) that delivered some of the advantages of the Hadoop stack. That said, Hadoop as an open source platform is much more sustainable technology. As long as software vendors can get past the decision to embrace using open source in their commercial products.

  4. Thanks for the comment!

    Well, Zions case may lean either way – they’ve been doing it for more than a decade and not many other orgs even started on the same journey.
    A lot of their stuff is VERY custom and specific to them. They may be early …or they may be special and thus alone.

    And, ironically, some of the 1gen SIEMs “think” that 10GBs is “big data” :-(

  5. Dan G says:

    I think Zions is going in the right direction, but I think they are at Level3 and just looking at reports and basic analytics. Where do you think they are on this Big Data Security Maturity Scale? http://bigsnarf.wordpress.com/2012/03/23/big-data-infosec-bigsnarf-open-source-solution/

  6. Hmmm, on that scale from what I can figure they are between 7 and 8, or maybe at 8+ in some parts of their operation. But then again, they have about a decade head start on the rest of the industry :-(



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.