Here is a tricky problem to solve: how do we compare technical threat intelligence (TI) feeds?
First, a quick definition is in order. While I comply with Gartner overall definition of Threat Intelligence, here I wanted to limit the discussion to technical (sometimes called “tactical” or “operational”) TI such as feeds of IPs, DNS names, URLs, MD5s, etc [and, yes, I am well-aware of the fact that purists consider such feeds to be “threat data” and not “threat intelligence”, but please don’t kick me for this here].
In any case, how to tell that a $200,000 list of “bad” IP addresses is better than a $0 list of “bad” IP addresses?
Let’s think about it together:
| Proposed measure | Ease of getting it | Usefulness for the TI user |
| Number of entries | Easy – just count’em | Debatable – is more better? Or noisier? |
| Certainty of entry badness | Moderate– providers may not know due to algorithms, blind aggregation, lack of context, etc | Important, but not sufficient; even proven badness does not convey relevance to your environment |
| Type of entry badness | Varied – some feeds only cover certain types (like C&C or exfiltration) | Useful; needed to consume the data effectively |
| Additional context data, extended schema fields, etc | Easy – look at the data to see what context is provided | Useful, but as an auxiliary |
| Update frequency | Easy – ask the vendor or check the data | Useful as long as it can be utilized at the same speed |
| Frequency of matches with your IT environment | Hard – requires operational usage of TI data | Yes, this is what makes the feed relevant and “actionable” |
| Frequency of matches in your environment NOT connected to an ongoing investigation | Hard – requires operational usage and an active IR team | The most useful; this is actionable in the best possible way (but impossible to know in advance) |
| Frequency of “false positives” | Hard – requires operational usage | Useful in combination with the above |
| Popularity | Medium – need to ask the provider or peers about who else uses the data | Somewhat useful, but requires detailed interviews on usage with peers for maximum value |
| Durability – continued use for detection over time | Hard – requires operational usage for a long time | Yes, this is what makes the feed not just actionable, but reliably actionable |
| Preprocessing by the provider | Medium – need to ask the provider and trust the answer | Sort of – presumably feed provider processing makes it “better”, but how? |
| Exclusivity | Hard – no trustworthy way of getting it | Yes, but needs to be combined with some relevance metrics – if the provider TI feed has unique threats that don’t matter to you |
Keep in mind that the above is very, very, very raw and needs lots of refinement if not rework.
To conclude, most useful TI feed comparison metrics [in this list] require operational usage and thus cannot be easily utilized for feed purchase/integration decision. In other words, I haven’t solved the problem yet.
Effectively, we need a metric that servers as a proxy for “how much will this TI feed reduce my time to detect?” (detect faster) and “how much will this TI feed enable me to detect what I’d otherwise miss?” (detect better).
Finally, this conundrum made some organization say “We’ll just collect *ALL* possible feeds and build a local intel clearing operation.” This approach treats all TI feeds as “raw threat data” and then focuses on creating locally relevant threat intel out of the pile. That certainly works well, if you have the resources to do it.
Ideas? Thoughts? Experiences?
Huge thanks to Lenny Zeltser for super-helpful comments on this emerging framework!
Posts related to this research project:
The Gartner Blog Network provides an opportunity for Gartner analysts to test ideas and move research forward. Because the content posted by Gartner analysts on this site does not undergo our standard editorial review, all comments or opinions expressed hereunder are those of the individual contributors and do not represent the views of Gartner, Inc. or its management.
Comments are closed
6 Comments
This is an interesting start and would love to have a conversation about this. I have been looking into several services and have found that many of them use similar sources of automatable data sources and they are providing access and an interface to that combined data, but each approaches the problem differently. Others focus on their ability to have their people find the data you want.
I have seen companies with deep pockets do the “collect it all and hire people to sort it out” but most companies can’t afford that, as you mention. I have seen more companies talking about having a dedicated person to just trying to find better sources and better ways of correlating and using the data.
Does this just feed into a discussion of “big data” and data mining? Do current tools have the ability to make this data useful or do you rely on the vendor/provider to identify what you need?
Lee, thanks for the comment.
> I have seen more companies talking about having a dedicated person
>to just trying to find better sources and better ways of correlating and
>using the data.
Well, the above point relies on knowing what BETTER sources really means. I ‘ll try to actually formalize the definition of that “better-ness” as it applies to TI feeds
>big data
It does, actually. “Collect it all and then cleanse the data” (at those that can afford it) often relies on Hadoop and other big data-ish components and approaches.
Problem is that many are trying to do it with SIEM products they already own (to check a box). And while the SIEM tools may start supporting a Hadoop backend, setting it up properly is a completely different issue. So does this make a TI with a UI rather than a feed more favorable to those with less budget?
On “certainty of badness” I have major concerns. I saw one vendor that marked the google open DNS resolver as the highest level of badness because what was talking to it but not because of what it is. I am not claiming it was right or wrong but that the method for scoring is potentially flawed.
How do you define better sources? Does more sources mean better correlation or more accurate ranking?
Hi Anton,
as briefly discussed on twitter, here are my thoughts on some of the categories you mention.
Type of entry badness : I would rather evaluate ‘variety of entry badness type’. This allows you to compare one list to the other and weigh on the variety they offer.
I don’t necessarily agree with your evaluation of ‘Update Frequency’. In my experience, feeds that are faster than you consume them are better than feeds that are slower. So a good feed would be a feed that can deliver data at a pace faster or equal to your consumption ability.
Finally, I looked at ‘Frequency of matches with your IT environment’ and its sibling with the longer name. This obviously makes sense because a feed needs to provide relevant and actionable data. I would however consider an ‘ability to customize with your IT environment’ criteria. Some intelligence feeds will allow you to do that (based on risk profile? based on technology footprint?) and it reduces the amount of data you need to filter and consume.
Cheers,
Wim
In the case of analysis this kind of data can help with some of the heavy lifting that makes analysis products work better out of the box (albeit in a potentially misguided way). In the case of blocking the consequences of these lists being flawed can create significant problems though in some cases staging and tightening (as per IPS signatures) can help.
@Lee Thanks for another insightful comment.
“TI with UI” is actually not a bad thing, whether you are in Hadoop land or in MySQL land. People lauched whole companies on search UIs for lots of TI sources.
>certainty of badness
For sure! This is indeed a nasty little problem with many feeds. Not just false positives, but in general what *really bad* means. The vendor may be very sure about it, but also wrong 🙂
>how do you define sources?
I didn’t – and it is starting to bite me in the ass 🙂 I really need to think more clearly about sources, types of sources, etc. Is a honeynet of 100 systems one source? Is rumor received over beers from a peer a source? Is SANS ISC data feed (combining many sources?) a source?
@Wim
Thanks for the comments. I agree with your point on update frequency (at >= of your process, with faster = better)
What about the ability to customize? Is it the same as “data formats provided” – XML, JSON, CSV, STIX, etc? Or does it mean “get all feed OR get filtered subset”?
@Matt
Exactly the point re: blocking vs analysis. Block-quality lists probably face way more requirements BUT due to those very restrictions may end up being less useful for real early warning about advanced threats. If you are 101% obsessed about FPs, you are bound to have nasty FNs…