You have 10 petabytes of security data in your Hadoop cluster.
You count RAM in terabytes and CPU cores in dozens.
You speak HiveQL better than you speak English.
You collect literally and unquestionably every timed record of activity in your organization – including transaction logs, IM messages, flows, anything.
You run queries over 13 months of data – and you do not have to take a vacation before the results come in.
You outgrew your market-leading SIEM product … 5 years ago.
You have statisticians (data scientists) on speed-dial – and on staff.
You run statistical models on volumes of security data before your morning coffee – and get good results.
Your organizations’ BI team thinks you are actually cool… despite being in security.
So….
are you a HARBINGER or an OUTLIER?
Is this the way information security will be done nearly everywhere in 3, 5, 10 years? (good arguments for this)
Or is this a case of “there are only 10 organizations in a Top 10 list”? (some arguments for this)
Is this the way we all need to learn to succeed with current and future threats?
Or is this the way to the top of the mountain that only the enlightened gurus will ever tread?
In any case, let’s keep this discussion going!
P.S. By the way, remember that: “If at first you don’t succeed, skydiving may not be for you.” [by unknown] –> “If you keep failing with small data now, BIG DATA isn‘t for you!” [by Anton Chuvakin]
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.
Comments are closed
5 Comments
I had a laugh reading this, ingenious! I think I can relate. Isn’t it great if we can predict a disaster from happening thousands of miles away? I think, I’ll be an outlier for now and observe the buzz around Big Data et al.
Probably a harbinger, based especially on the leading work done by Zions Bank using this technology stack, people competencies and analytic methods.
Wider adoption will come from maturation of the technology stack that will reduce the need for such specialized skills and level of effort. But the basic appeal of Big Data for SIEM/log management security remains: huge data volumes, ability to hold disparate data sets of varied formats, relatively cheap processing, etc.
ironically, some of the first generation of SIEMs and log management products were built on proprietary file management systems (e.g. Network Intelligence/RSA enVision) that delivered some of the advantages of the Hadoop stack. That said, Hadoop as an open source platform is much more sustainable technology. As long as software vendors can get past the decision to embrace using open source in their commercial products.
Thanks for the comment!
Well, Zions case may lean either way – they’ve been doing it for more than a decade and not many other orgs even started on the same journey.
A lot of their stuff is VERY custom and specific to them. They may be early …or they may be special and thus alone.
And, ironically, some of the 1gen SIEMs “think” that 10GBs is “big data” 🙁
I think Zions is going in the right direction, but I think they are at Level3 and just looking at reports and basic analytics. Where do you think they are on this Big Data Security Maturity Scale? http://bigsnarf.wordpress.com/2012/03/23/big-data-infosec-bigsnarf-open-source-solution/
Hmmm, on that scale from what I can figure they are between 7 and 8, or maybe at 8+ in some parts of their operation. But then again, they have about a decade head start on the rest of the industry 🙁