Is there bound to be hype? Absolutely. That’s why Gartner publishes a large number of technology hype cycles each year. Technologies invariably get overhyped, fall into the “Trough of Disillusionment” and ultimately assume an important role in our IT infrastructure (“The Plateau of Productivity”).
No doubt “big data” is heading towards the peak of inflated expectations (including its role in information security) but that doesn’t mean there aren’t tough information security problems that would benefit from big data analytics being applied to specific information security problems. In fact, I’d argue that most information security providers are already performing what would be characterized as big data analytics processing on their back-ends to process the large numbers of events, IP addresses, URLs, files and other attributes used to identify and track threats.
Here’s a couple key points to consider:
1) While the labs of Symantec, Trend, McAfee, Sophos, Microsoft, Sourcefire, Check Point, etc etc will be performing big data analytics on our behalf on their back end, they don’t necessarily have detailed monitoring of our own enterprise network and systems (packet data, flow data, sessions, transactions and so on). The need for internal monitoring and big data analytics against this will be a cornerstone of our strategies to detect advanced targeted attacks that have bypassed traditional protection mechanisms (e.g. anomaly detection).
2) Big Data projects can’t be solely about the data, they have to be focused on the timely deliver of insight and intelligence derived from this data.
3) Although most “big data” discussions focus on the volume of data as being the key attribute of being “big”, there are three other attributes that also define “bigness” – variety, velocity and complexity. This is especially true for information security where some decisions are needed in near real time from a large variety and velocity of data in a relatively short period of time (think SIEM). Here, the raw volume of the data is relatively small, it’s the need to process this in real time/near-real time (typically in memory) that makes this a big data type of problem.
4) “Big Data” isn’t new per se. What’s changed is that Moore’s Law, 64 bit processors, in memory databases, scale-out software architectures, open source technologies like Hadoop, MapReduce, Cassandra and so on have lowered the barrier to entry for all enterprises. Big Data isn’t new, but it is likely new to your organization — including its role in information security.
5) Contrary to what the SIEM vendors might say – a) they aren’t necessarily the “heir apparent” to the big data analytics role, especially for after the fact data mining and b) correlation and analytics will not soley be the domain of SIEM
6) While so far I have written about the use of big data analytics to solve the next-generation of security problems, we can not and must not ignore the need to secure these big data assets – information security, business intelligence or otherwise. Open source scale-out architectures such as Hadoop are cool, but like most new technologies, security is an afterthought. The biggest holes? authentication, authorization, role based access control, auditing, monitoring, backup and recovery and so on . You know, security 101.
Don’t fall for the hype, but at the same time don’t dismiss big data and information security as a passing fad. It isn’t.
Big data analytics will be absolutely foundational to solving the next-generation of tough information security problems.
I’ve outlined the current state of the market in this research note for clients titled “Information Security is Becoming a Big Data Analytics Problem”
I’ll also be presenting on this topic at Gartner’s Information Security Summit next month in Washington DC. I hope to see you there.
Category: Beyond Anti-Virus Big Data and Information Security Next-generation Security Infrastructure Security Intelligence Tags: Adaptive Security Infrastucture, Beyond Anti-Virus, Big Data, Next-generation Security Infrastructure, Security-Summit-NA