Gartner Blog Network

Information Security and Big Data–Hype or Hope?

by Neil MacDonald  |  May 22, 2012  |  6 Comments

I been a proponent of the use of big data analytics techniques being applied to the next-generation of information security problems.

Is there bound to be hype? Absolutely. That’s why Gartner publishes a large number of technology hype cycles each year. Technologies invariably get overhyped, fall into the “Trough of Disillusionment” and ultimately assume an important role in our IT infrastructure (“The Plateau of Productivity”).



No doubt “big data” is heading towards the peak of inflated expectations (including its role in information security) but that doesn’t mean there aren’t tough information security problems that would benefit from big data analytics being applied to specific information security problems. In fact, I’d argue that most information security providers are already performing what would be characterized as big data analytics processing on their back-ends to process the large numbers of events, IP addresses, URLs, files and other attributes used to identify and track threats.

Here’s a couple key points to consider:

1) While the labs of Symantec, Trend, McAfee, Sophos, Microsoft, Sourcefire, Check Point, etc etc will be performing big data analytics on our behalf on their back end, they don’t necessarily have detailed monitoring of our own enterprise network and systems (packet data, flow data, sessions, transactions and so on). The need for internal monitoring and big data analytics against this will be a cornerstone of our strategies to detect advanced targeted attacks that have bypassed traditional protection mechanisms (e.g. anomaly detection).

2) Big Data projects can’t be solely about the data, they have to be focused on the timely deliver of insight and intelligence derived from this data.

3) Although most “big data” discussions focus on the volume of data as being the key attribute of being “big”, there are three other attributes that also define “bigness” – variety, velocity and complexity. This is especially true for information security where some decisions are needed in near real time from a large variety and velocity of data in a relatively short period of time (think SIEM). Here, the raw volume of the data is relatively small, it’s the need to process this in real time/near-real time (typically in memory) that makes this a big data type of problem.

4) “Big Data” isn’t new per se. What’s changed is that Moore’s Law, 64 bit processors, in memory databases, scale-out software architectures, open source technologies like Hadoop, MapReduce, Cassandra and so on have lowered the barrier to entry for all enterprises. Big Data isn’t new, but it is likely new to your organization —  including its role in information security.

5) Contrary to what the SIEM vendors might say – a) they aren’t necessarily the “heir apparent” to the big data analytics role, especially for after the fact data mining and b) correlation and analytics will not soley be the domain of SIEM

6) While so far I have written about the use of big data analytics to solve the next-generation of security problems, we can not and must not ignore the need to secure these big data assets – information security, business intelligence or otherwise. Open source scale-out architectures such as Hadoop are cool, but like most new technologies, security is an afterthought. The biggest holes? authentication, authorization, role based access control, auditing, monitoring, backup and recovery and so on . You know, security 101.


Don’t fall for the hype, but at the same time don’t dismiss big data and information security as a passing fad. It isn’t.

Big data analytics will be absolutely foundational to solving the next-generation of tough information security problems.

I’ve outlined the current state of the market in this research note for clients titled “Information Security is Becoming a Big Data Analytics Problem”

I’ll also be presenting on this topic at Gartner’s Information Security Summit next month in Washington DC. I hope to see you there.

Category: beyond-anti-virus  big-data-and-information-security  next-generation-security-infrastructure  security-intelligence  

Tags: adaptive-security-infrastucture  beyond-anti-virus  big-data  next-generation-security-infrastructure  security-summit-na  

Neil MacDonald
VP & Gartner Fellow
15 years at Gartner
25 years IT industry

Neil MacDonald is a vice president, distinguished analyst and Gartner Fellow in Gartner Research. Mr. MacDonald is a member of Gartner's information security and privacy research team, focusing on operating system and application-level security strategies. Specific research areas include Windows security…Read Full Bio

Thoughts on Information Security and Big Data–Hype or Hope?

  1. Andre Gironda says:

    Your number 5 and number 1 are the same?

    My thoughts — total intelligence is total intelligence. Remember the \Know your Enemy\ series?

    Big data can help, but when adversaries are also using these intelligence techniques to aid in their strategies and attack paths, then it all comes down to who implemented a better system for the weak point that is being contended at the time.

    Another point I need to make here is that sharing alliances still need to open or trade data in order to make it useful when performing intelligence on bad actors that target that specific shared industry. The adversaries of the banking and payment card industries have often been found to be sharing information, as an example, which initially placed these industries at a critical weakness.

  2. Emil Glownia says:

    Great graph that is what I thought but the graph shows it very well :) It will get handy

  3. Neil MacDonald says:


    Oops – cut and paste user error

    5) Contrary to what the SIEM vendors might say – a) they aren’t necessarily the “heir apparent” to the big data analytics role, especially for after the fact data mining and b) correlation and analytics will not soley be the domain of SIEM

    As to your point on sharing of data – I agree. Why do the bad guys (in general) do a better job of sharing intelligence than we do? Threat intelligence should be shared at least in industry-specific enclaves, across enterprises and across vendor stacks – e.g. Symantec, McAfee, Sourcefire, etc


  4. Neil MacDonald says:

    All- I updated the post with the corrected #5

  5. Mark Seward says:

    In recent discussions with customers, they are telling us that big-data changes the nature of their jobs. Learning and using math skills to understand attacker profiles is starting to (without their knowing it) turn security practitioners into data scientists — and they love it!

  6. Neil MacDonald says:


    The next 10 years will be exciting for information security professionals and the skill sets will change.

    Here’s what I said in this research note for clients:

    “By 2016, 40% of Type A enterprises will create and staff a security analytics role, up from less than
    1% in 2011.”

    I agree with you, even without the title – security data scientists / security analytics is the futre.


Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.