One more case that occasionally (not as often as Case 1) shows up is “data variety explosion.”
Specifically, this scenario goes like this:
- The organization gets a SIEM tool and generally likes it (a good SIEM can be effectively used for both real-time and historical data analysis – typically in the 14-90 days range)
- Use cases are defined, logs and context data collected to support them
- Their use of SIEM expands, and the organization starts to think what else they can consume and analyze
- Some people suggest flows, others – packets; there is a group that wants email contents, IM conversations, application transaction records, surfed web pages, transmitted files, system configurations; then somebody suggests phone call records, social status feeds , voice recordings and eventually video from cameras …
- In essence, a dizzying array of timed records and loosely related context data is proposed to solve various problems (often related to not just infosec, but also non-IT compliance, fraud, safety, operational risk, etc)
- The organization looks beyond SIEM and discovers the wonderful world of Hadoop, Cassandra, MongoDB and other “big data-ish” technologies and starts their own big data project.
What is the purpose of this “high variety” pile of data? Linking and relating! As a result, the organization may [eventually] possess a central data hub, useful for wide array of projects, going much beyond infosec and IT. The system enables all sorts of analysis of data relationships, whether starting from logs/alerts or not.
Related posts on the topic of big data for security:
- Big Data for Security Realities: Case 1: Too Much Volume To Store aka “Big Data Collection”
- Big Data Analytics for Security: Having a Goal + Exploring
- More On Big Data Security Analytics Readiness
- Broadening Big Data Definition Leads to Security Idiotics!
- Next Research Project: From Big Data Analytics to … Patching
- 9 Reasons Why Building A Big Data Security Analytics Tool Is Like Building a Flying Car
- “Big Analytics” for Security: A Harbinger or An Outlier?
- All posts tagged big data