Big Data for Security Realities – Case 2 Variety Explosion

Part of my research this quarter focuses on assessing the reality of using big data approaches for security and providing practical, GTP-style recommendations for enterprises. So, what else IS real?

One more case that occasionally (not as often as Case 1) shows up is “data variety explosion.”

Specifically, this scenario goes like this:

1. The organization gets a SIEM tool and generally likes it (a good SIEM can be effectively used for both real-time and historical data analysis – typically in the 14-90 days range)
2. Use cases are defined, logs and context data collected to support them
3. Their use of SIEM expands, and the organization starts to think what else they can consume and analyze
4. Some people suggest flows, others – packets; there is a group that wants email contents, IM conversations, application transaction records, surfed web pages, transmitted files, system configurations; then somebody suggests phone call records, social status feeds , voice recordings and eventually video from cameras …
5. In essence, a dizzying array of timed records and loosely related context data is proposed to solve various problems (often related to not just infosec, but also non-IT compliance, fraud, safety, operational risk, etc)
6. The organization looks beyond SIEM and discovers the wonderful world of Hadoop, Cassandra, MongoDB and other “big data-ish” technologies and starts their own big data project.

What is the purpose of this “high variety” pile of data? Linking and relating! As a result, the organization may [eventually] possess a central data hub, useful for wide array of projects, going much beyond infosec and IT. The system enables all sorts of analysis of data relationships, whether starting from logs/alerts or not.

Related posts on the topic of big data for security:

• Big Data for Security Realities: Case 1: Too Much Volume To Store aka “Big Data Collection”
• Big Data Analytics for Security: Having a Goal + Exploring
• More On Big Data Security Analytics Readiness
• Broadening Big Data Definition Leads to Security Idiotics!
• Next Research Project: From Big Data Analytics to … Patching
• 9 Reasons Why Building A Big Data Security Analytics Tool Is Like Building a Flying Car
• “Big Analytics” for Security: A Harbinger or An Outlier?
• All posts tagged big data