Part of my research this quarter focuses on assessing the reality of using big data approaches for security and providing practical, GTP-style recommendations for enterprises. So, what else is real in this technology segment heavily overrun by waves of bull?
One more case that occasionally show up is “Big But Narrowly Used Data.”
The scenario may go like this:
- The organization comes across a need to analyze a particular large data set (such as 50-500GB of web proxy logs) for a particular goal (say match accessed URLs to a set of blacklist) [back in the old, “pre-big data” days, I’ve met somebody with a trillion log messages stashed in an old sock somewhere]
- An initial attempt to load the data set into a SIEM or a log management tool fails since there is no existing capacity for such a volume and no new capacity is approved.
- Some at an organization tries to “brute force”: writes a Perl script to do this on the flat files. Days of impatient waiting ensue 🙂
- Suddenly somebody thinks: Hadoop!
- The cluster is put together and the data loaded and analysis queries written.
- And this is the step where the magic may or may not happen: the organization may decide to use the same approach for other data-intensive security problems and therefore starts on the road of using big data for security…
There you have it!
Related posts on the topic of big data for security:
- Gartner Predicts 2014 Out – And One On Big Data!
- Big Data Analytics Mindset – What Is It?
- Big Data for Security Realities – Case 3: Elastic Search or Similar
- Big Data for Security Realities – Case 2 Variety Explosion
- Big Data for Security Realities: Case 1: Too Much Volume To Store aka “Big Data Collection”
- Big Data Analytics for Security: Having a Goal + Exploring
- More On Big Data Security Analytics Readiness
- Broadening Big Data Definition Leads to Security Idiotics!
- Next Research Project: From Big Data Analytics to … Patching
- 9 Reasons Why Building A Big Data Security Analytics Tool Is Like Building a Flying Car
- “Big Analytics” for Security: A Harbinger or An Outlier?
- All posts tagged big data