by Anton Chuvakin | October 17, 2013 | Comments Off on Big Data for Security Realities – Case 2 Variety Explosion
One more case that occasionally (not as often as Case 1) shows up is “data variety explosion.”
Specifically, this scenario goes like this:
- The organization gets a SIEM tool and generally likes it (a good SIEM can be effectively used for both real-time and historical data analysis – typically in the 14-90 days range)
- Use cases are defined, logs and context data collected to support them
- Their use of SIEM expands, and the organization starts to think what else they can consume and analyze
- Some people suggest flows, others – packets; there is a group that wants email contents, IM conversations, application transaction records, surfed web pages, transmitted files, system configurations; then somebody suggests phone call records, social status feeds , voice recordings and eventually video from cameras …
- In essence, a dizzying array of timed records and loosely related context data is proposed to solve various problems (often related to not just infosec, but also non-IT compliance, fraud, safety, operational risk, etc)
- The organization looks beyond SIEM and discovers the wonderful world of Hadoop, Cassandra, MongoDB and other “big data-ish” technologies and starts their own big data project.
What is the purpose of this “high variety” pile of data? Linking and relating! As a result, the organization may [eventually] possess a central data hub, useful for wide array of projects, going much beyond infosec and IT. The system enables all sorts of analysis of data relationships, whether starting from logs/alerts or not.
Related posts on the topic of big data for security:
- Big Data for Security Realities: Case 1: Too Much Volume To Store aka “Big Data Collection”
- Big Data Analytics for Security: Having a Goal + Exploring
- More On Big Data Security Analytics Readiness
- Broadening Big Data Definition Leads to Security Idiotics!
- Next Research Project: From Big Data Analytics to … Patching
- 9 Reasons Why Building A Big Data Security Analytics Tool Is Like Building a Flying Car
- “Big Analytics” for Security: A Harbinger or An Outlier?
- All posts tagged big data
100 Data and Analytics Predictions Through 2024
Gartner’s annual predictions disclose the varied importance of data and analytics across an ever-widening range of business and IT initiatives. Data and analytics leaders must consider these strategic planning assumptions for enhancing their vision and plans.Read Free Gartner Research
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.