by Anton Chuvakin | May 1, 2018 | Comments Off on Why POCs Fail and Why You Must POC Anyway!?
A lot of people in the industry assume that we Gartner analysts walk on water …
… and we do. We do walk on the churning waters of vendor propaganda, misdirection and “messaging.”
However, sometimes when clients ask us a tough question about how well some technology will work in their environment, we tell them “TEST IT FOR YOURSELF.” In other words, do a proof of concept deployment (PoC / POC). This post is my brief lament about the glamors and horrors of today’s security technology POC.
Now, most if not all of our recent papers (on SIEM, SaaS SIEM, UEBA, SOAR, deception, etc) say something like “conduct a 14-30 day POC deployment.” For example, our recent UEBA piece says “although a common POC duration is 30 days, much longer POC deployments are not uncommon [for UEBA].”
Why is that? Here are some ideas:
- For some technologies, such as those that use various non-deterministic approaches (today this is a wide range from EPP to EDR to UEBA), your choice is “POC-based or luck-based” and no other (due to this stuff, mostly). In the 1990s, you can count the signatures of 2 IDS devices, but with today’s “ML-rich” security tech, you just have to POC.
- Make POC as close to production as possible. We did see sad stories of “POC win, production fail” (in particular, for UEBA) where the test relied on assumptions that were just plain incorrect in customer’s own production environment, or the POC was “hand-tweaked” by the field engineers who then left and the tool fell apart.
- Appreciate “PoC gems” (amazing, but likely totally random finds during the POC, like being hacked by the elite APT at the very moment of a new tech test), but don’t rely solely on them to buy. This may never happen again! Understand why they were detected and what else can be – in your environment.
- You may not have enough people or time to do a POC. What is the solution in this case? Do a POC anyway – or waste money on tools that will not work for you. Sorry, but this is the truth.
For additional advice, read the section “Running a UEBA POC” of our “A Comparison of UEBA Technologies and Solutions” (Gartner GTP access required) or an excellent “Use the Gartner Playbook for a Successful SIEM Proof of Concept” (Gartner access required).
Read Complimentary Relevant Research
How to Evaluate Cloud Service Provider Security
Security and risk management leaders continue to experience challenges to efficiently and reliably determine whether cloud service providers...
View Relevant Webinars
Managing the Security Risks of IoT Innovation
While IoT provides a revolutionary approach to technology innovation for digital business, with great change comes great responsibility....
Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.