Gartner Blog Network


The False Positive Problem in Anti-Spam: Peter Firstbrook

by Greg Young  |  March 10, 2010  |  2 Comments

Guest blog by Peter Firstbrook

While doing the research for our forthcoming secure email gateway Magic Quadrant, we are very disappointed with how few anti-spam solutions have reports that show the false positive (legit email tagged as spam) and false negative (spam that get to the inbox) rates.  While there is no perfect way to measure spam accuracy exactly there are good proxies that can be easily measured; false positives can be represented by the emails that were released from quarantines, and false negatives are messages that make it into the inbox that users reported as spam.  (And while we are at it, shame on any anti-spam solution that does not even offer an email client “is Spam” button.) Do anti-spam solutions have something to hide?  Most brag about their “honeypot” catch rates but “honeypots” rarely get legitimate email.

Although Gartner customers almost never complain about false positive rates, I wonder if false positives are under estimated. End users rarely complain about false positives, but they are very vocal reporting Spam in their inbox. Box Sentry (www.boxsentry.com) recently did a tests in a number of organizations and found the false positive rate in some organizations using popular ant-spam tools was as high as 13% of legitimate emails. The largest proportion of false positives in their study was legitimate person-to-person traffic.  While it could be that these organizations have over-tuned their systems to block more Spam at the expense of quarantining more legit email, the reality was the email administrators had no idea they had such a high false positive rate because they never checked. Have you?  Organizations that do not send daily digests to end users should check their quarantine to ensure that it is not a tar pit of business critical communications.   Let us know what you find.

Peter Firstbrook| Research Director| Gartner
Malware and antispam

Category: 

Greg Young
Research VP
6 years at Gartner
22 years IT security

Greg Young is a research vice president in Gartner and the lead analyst for network security. Mr. Young has experience in IT security in product companies, and in both the private and public sectors. He spent his military career in technology security… Read Full Bio


Thoughts on The False Positive Problem in Anti-Spam: Peter Firstbrook


  1. […] a good post at Gartner pointing out the lack of data reported by vendors or customers regarding the false […]

  2. Hi Peter

    I agree that the reporting in most solutions are pretty poor – it was one of the main reasons for partnering with Preserv8. Your post did prompt me to go and review our reports a little closer. In March so far 0.6% mails were initially marked as Spam. Of those I am sure that probably 5 of them were false positive. I must point out that in march I cannot recall 1 Spam email getting through to my Inbox.

    Also Preserv8’s perimeter defence deleted the following on the perimeter
    9410 mails with no valid recipient
    38166 mails originating from know spamming servers
    729 mails from servers with Outbound relaying denies
    134 mails from blacklisted email addresses

    I find that my false positives are caused by signing up for a new newsletter as the domian / email is not recorded within my ever growing whitelist.

    Whilst there is not a report in Preserv8’s growing list to cover false positives they are normally very open about generating new reports for users and adding it to the standard reporting list.

    Regards

    Lawrence



Comments are closed

Comments or opinions expressed on this blog are those of the individual contributors only, and do not necessarily represent the views of Gartner, Inc. or its management. Readers may copy and redistribute blog postings on other blogs, or otherwise for private, non-commercial or journalistic purposes, with attribution to Gartner. This content may not be used for any other purposes in any other formats or media. The content on this blog is provided on an "as-is" basis. Gartner shall not be liable for any damages whatsoever arising out of the content or use of this blog.