Anton Chuvakin

A member of the Gartner Blog Network

Anton Chuvakin
Research VP
2+ years with Gartner
14 years IT industry

Anton Chuvakin is a research VP at Gartner's GTP Security and Risk Management group. Before Mr. Chuvakin joined Gartner, his job responsibilities included security product management, evangelist… Read Full Bio

Coverage Areas:

On Comparing Threat Intelligence Feeds

by Anton Chuvakin  |  January 7, 2014  |  7 Comments

Here is a tricky problem to solve: how do we compare technical threat intelligence (TI) feeds?

First, a quick definition is in order. While I comply with Gartner overall definition of Threat Intelligence, here I wanted to limit the discussion to technical (sometimes called “tactical” or “operational”) TI such as feeds of IPs, DNS names, URLs, MD5s, etc [and, yes, I am well-aware of the fact that purists consider such feeds to be “threat data” and not “threat intelligence”, but please don’t kick me for this here].

In any case, how to tell that a $200,000 list of “bad” IP addresses is better than a $0 list of “bad” IP addresses?

Let’s think about it together:

Proposed measure Ease of getting it Usefulness for the TI user
Number of entries Easy – just count’em Debatable – is more better? Or noisier?
Certainty of entry badness Moderate– providers may not know due to algorithms, blind aggregation, lack of context, etc Important, but not sufficient; even proven badness does not convey relevance to your environment
Type of entry badness Varied – some feeds only cover certain types (like C&C or exfiltration) Useful; needed to consume the data effectively
Additional context data, extended schema fields, etc Easy – look at the data to see what context is provided Useful, but as an auxiliary
Update frequency Easy – ask the vendor or check the data Useful as long as it can be utilized at the same speed
Frequency of matches with your IT environment Hard – requires operational usage of TI data Yes, this is what makes the feed relevant and “actionable”
Frequency of matches in your environment NOT connected to an ongoing investigation Hard – requires operational usage and an active IR team The most useful; this is actionable in the best possible way (but impossible to know in advance)
Frequency of “false positives” Hard – requires operational usage Useful in combination with the above
Popularity Medium – need to ask the provider or peers about who else uses the data Somewhat useful, but requires detailed interviews on usage with peers for maximum value
Durability – continued use for detection over time Hard – requires operational usage for a long time Yes, this is what makes the feed not just actionable, but reliably actionable
Preprocessing by the provider Medium – need to ask the provider and trust the answer Sort of – presumably feed provider processing makes it “better”, but how?
Exclusivity Hard – no trustworthy way of getting it Yes, but needs to be combined with some relevance metrics – if the provider TI feed has unique threats that don’t matter to you

Keep in mind that the above is very, very, very raw and needs lots of refinement if not rework.

To conclude, most useful TI feed comparison metrics [in this list] require operational usage and thus cannot be easily utilized for feed purchase/integration decision. In other words, I haven’t solved the problem yet.

Effectively, we need a metric that servers as a proxy for “how much will this TI feed reduce my time to detect?” (detect faster) and “how much will this TI feed enable me to detect what I’d otherwise miss?” (detect better).

Finally, this conundrum made some organization say “We’ll just collect *ALL* possible feeds and build a local intel clearing operation.” This approach treats all TI feeds as “raw threat data” and then focuses on creating locally relevant threat intel out of the pile. That certainly works well, if you have the resources to do it.

Ideas? Thoughts? Experiences?

Huge thanks to Lenny Zeltser for super-helpful comments on this emerging framework!

Posts related to this research project:

7 Comments »

Category: security threat intelligence     Tags:

7 responses so far ↓

  • 1 Lee Heath   January 7, 2014 at 8:59 pm

    This is an interesting start and would love to have a conversation about this. I have been looking into several services and have found that many of them use similar sources of automatable data sources and they are providing access and an interface to that combined data, but each approaches the problem differently. Others focus on their ability to have their people find the data you want.

    I have seen companies with deep pockets do the “collect it all and hire people to sort it out” but most companies can’t afford that, as you mention. I have seen more companies talking about having a dedicated person to just trying to find better sources and better ways of correlating and using the data.

    Does this just feed into a discussion of “big data” and data mining? Do current tools have the ability to make this data useful or do you rely on the vendor/provider to identify what you need?

  • 2 Anton Chuvakin   January 7, 2014 at 9:27 pm

    Lee, thanks for the comment.

    > I have seen more companies talking about having a dedicated person
    >to just trying to find better sources and better ways of correlating and
    >using the data.

    Well, the above point relies on knowing what BETTER sources really means. I ‘ll try to actually formalize the definition of that “better-ness” as it applies to TI feeds

    >big data

    It does, actually. “Collect it all and then cleanse the data” (at those that can afford it) often relies on Hadoop and other big data-ish components and approaches.

  • 3 Lee Heath   January 8, 2014 at 3:29 am

    Problem is that many are trying to do it with SIEM products they already own (to check a box). And while the SIEM tools may start supporting a Hadoop backend, setting it up properly is a completely different issue. So does this make a TI with a UI rather than a feed more favorable to those with less budget?

    On “certainty of badness” I have major concerns. I saw one vendor that marked the google open DNS resolver as the highest level of badness because what was talking to it but not because of what it is. I am not claiming it was right or wrong but that the method for scoring is potentially flawed.

    How do you define better sources? Does more sources mean better correlation or more accurate ranking?

  • 4 Wim Remes   January 8, 2014 at 8:38 am

    Hi Anton,

    as briefly discussed on twitter, here are my thoughts on some of the categories you mention.

    Type of entry badness : I would rather evaluate ‘variety of entry badness type’. This allows you to compare one list to the other and weigh on the variety they offer.

    I don’t necessarily agree with your evaluation of ‘Update Frequency’. In my experience, feeds that are faster than you consume them are better than feeds that are slower. So a good feed would be a feed that can deliver data at a pace faster or equal to your consumption ability.

    Finally, I looked at ‘Frequency of matches with your IT environment’ and its sibling with the longer name. This obviously makes sense because a feed needs to provide relevant and actionable data. I would however consider an ‘ability to customize with your IT environment’ criteria. Some intelligence feeds will allow you to do that (based on risk profile? based on technology footprint?) and it reduces the amount of data you need to filter and consume.

    Cheers,
    Wim

  • 5 Matt   January 8, 2014 at 3:59 pm

    In the case of analysis this kind of data can help with some of the heavy lifting that makes analysis products work better out of the box (albeit in a potentially misguided way). In the case of blocking the consequences of these lists being flawed can create significant problems though in some cases staging and tightening (as per IPS signatures) can help.

  • 6 Anton Chuvakin   January 8, 2014 at 8:33 pm

    @Lee Thanks for another insightful comment.

    “TI with UI” is actually not a bad thing, whether you are in Hadoop land or in MySQL land. People lauched whole companies on search UIs for lots of TI sources.

    >certainty of badness

    For sure! This is indeed a nasty little problem with many feeds. Not just false positives, but in general what *really bad* means. The vendor may be very sure about it, but also wrong :-)

    >how do you define sources?

    I didn’t – and it is starting to bite me in the ass :-) I really need to think more clearly about sources, types of sources, etc. Is a honeynet of 100 systems one source? Is rumor received over beers from a peer a source? Is SANS ISC data feed (combining many sources?) a source?

    @Wim

    Thanks for the comments. I agree with your point on update frequency (at >= of your process, with faster = better)

    What about the ability to customize? Is it the same as “data formats provided” – XML, JSON, CSV, STIX, etc? Or does it mean “get all feed OR get filtered subset”?

    @Matt

    Exactly the point re: blocking vs analysis. Block-quality lists probably face way more requirements BUT due to those very restrictions may end up being less useful for real early warning about advanced threats. If you are 101% obsessed about FPs, you are bound to have nasty FNs…

  • 7 The Conundrum of Two Intelligences!   January 15, 2014 at 9:05 pm

    [...] On Comparing Threat Intelligence Feeds [...]