Neil MacDonald

A member of the Gartner Blog Network

Neil MacDonald
VP & Gartner Fellow
15 years at Gartner
25 years IT industry

Neil MacDonald is a vice president, distinguished analyst and Gartner Fellow in Gartner Research. Mr. MacDonald is a member of Gartner's information security and privacy research team, focusing on operating system and application-level security strategies. Specific research areas include Windows security…Read Full Bio

Coverage Areas:

Information Security and Big Data–Hype or Hope?

by Neil MacDonald  |  May 22, 2012  |  6 Comments

I been a proponent of the use of big data analytics techniques being applied to the next-generation of information security problems.

Is there bound to be hype? Absolutely. That’s why Gartner publishes a large number of technology hype cycles each year. Technologies invariably get overhyped, fall into the “Trough of Disillusionment” and ultimately assume an important role in our IT infrastructure (“The Plateau of Productivity”).



No doubt “big data” is heading towards the peak of inflated expectations (including its role in information security) but that doesn’t mean there aren’t tough information security problems that would benefit from big data analytics being applied to specific information security problems. In fact, I’d argue that most information security providers are already performing what would be characterized as big data analytics processing on their back-ends to process the large numbers of events, IP addresses, URLs, files and other attributes used to identify and track threats.

Here’s a couple key points to consider:

1) While the labs of Symantec, Trend, McAfee, Sophos, Microsoft, Sourcefire, Check Point, etc etc will be performing big data analytics on our behalf on their back end, they don’t necessarily have detailed monitoring of our own enterprise network and systems (packet data, flow data, sessions, transactions and so on). The need for internal monitoring and big data analytics against this will be a cornerstone of our strategies to detect advanced targeted attacks that have bypassed traditional protection mechanisms (e.g. anomaly detection).

2) Big Data projects can’t be solely about the data, they have to be focused on the timely deliver of insight and intelligence derived from this data.

3) Although most “big data” discussions focus on the volume of data as being the key attribute of being “big”, there are three other attributes that also define “bigness” – variety, velocity and complexity. This is especially true for information security where some decisions are needed in near real time from a large variety and velocity of data in a relatively short period of time (think SIEM). Here, the raw volume of the data is relatively small, it’s the need to process this in real time/near-real time (typically in memory) that makes this a big data type of problem.

4) “Big Data” isn’t new per se. What’s changed is that Moore’s Law, 64 bit processors, in memory databases, scale-out software architectures, open source technologies like Hadoop, MapReduce, Cassandra and so on have lowered the barrier to entry for all enterprises. Big Data isn’t new, but it is likely new to your organization —  including its role in information security.

5) Contrary to what the SIEM vendors might say – a) they aren’t necessarily the “heir apparent” to the big data analytics role, especially for after the fact data mining and b) correlation and analytics will not soley be the domain of SIEM

6) While so far I have written about the use of big data analytics to solve the next-generation of security problems, we can not and must not ignore the need to secure these big data assets – information security, business intelligence or otherwise. Open source scale-out architectures such as Hadoop are cool, but like most new technologies, security is an afterthought. The biggest holes? authentication, authorization, role based access control, auditing, monitoring, backup and recovery and so on . You know, security 101.


Don’t fall for the hype, but at the same time don’t dismiss big data and information security as a passing fad. It isn’t.

Big data analytics will be absolutely foundational to solving the next-generation of tough information security problems.

I’ve outlined the current state of the market in this research note for clients titled “Information Security is Becoming a Big Data Analytics Problem”

I’ll also be presenting on this topic at Gartner’s Information Security Summit next month in Washington DC. I hope to see you there.


Category: Beyond Anti-Virus Big Data and Information Security Next-generation Security Infrastructure Security Intelligence     Tags: , , , ,

6 responses so far ↓

  • 1 Andre Gironda   May 22, 2012 at 8:11 pm

    Your number 5 and number 1 are the same?

    My thoughts — total intelligence is total intelligence. Remember the \Know your Enemy\ series?

    Big data can help, but when adversaries are also using these intelligence techniques to aid in their strategies and attack paths, then it all comes down to who implemented a better system for the weak point that is being contended at the time.

    Another point I need to make here is that sharing alliances still need to open or trade data in order to make it useful when performing intelligence on bad actors that target that specific shared industry. The adversaries of the banking and payment card industries have often been found to be sharing information, as an example, which initially placed these industries at a critical weakness.

  • 2 Emil Glownia   May 23, 2012 at 1:25 am

    Great graph that is what I thought but the graph shows it very well :) It will get handy

  • 3 Neil MacDonald   May 23, 2012 at 12:42 pm


    Oops – cut and paste user error

    5) Contrary to what the SIEM vendors might say – a) they aren’t necessarily the “heir apparent” to the big data analytics role, especially for after the fact data mining and b) correlation and analytics will not soley be the domain of SIEM

    As to your point on sharing of data – I agree. Why do the bad guys (in general) do a better job of sharing intelligence than we do? Threat intelligence should be shared at least in industry-specific enclaves, across enterprises and across vendor stacks – e.g. Symantec, McAfee, Sourcefire, etc


  • 4 Neil MacDonald   May 24, 2012 at 10:28 am

    All- I updated the post with the corrected #5

  • 5 Mark Seward   May 29, 2012 at 12:15 pm

    In recent discussions with customers, they are telling us that big-data changes the nature of their jobs. Learning and using math skills to understand attacker profiles is starting to (without their knowing it) turn security practitioners into data scientists — and they love it!

  • 6 Neil MacDonald   May 29, 2012 at 1:04 pm


    The next 10 years will be exciting for information security professionals and the skill sets will change.

    Here’s what I said in this research note for clients:

    “By 2016, 40% of Type A enterprises will create and staff a security analytics role, up from less than
    1% in 2011.”

    I agree with you, even without the title – security data scientists / security analytics is the futre.