Patch Management – NOT A Solved Problem!

We again interrupt our regular programming (on network forensics and security data sharing this quarter) to delve into a subject much removed from the exciting world of APT fighting, “kill chain” swinging, state-sponsored attacking and stealthy custom malware-ing. My esteemed colleagues (like here) like to sometimes use the labels of “security haves” vs “security have-nots.”

Hereby we are transported into the land of the unfortunate “have-nots” from the land of the enlightened “haves.”

Although  patching has been “a solved problem” for many years, even decades, a lot of organizations struggle with it today – and struggle mightily.  While patching Windows OS on desktops monthly is indeed solved everywhere, but in the darkest woods of IT, patching 3rd party application on a desktop remains a significant challenge for many organizations. Patching server OSs (Windows and Linux/UNIX) and 3rd party server applications also remains challenging due to fragility of many server environments. Add virtualization to the mix – and you have a full-blown slow-cooking disaster. And then you have Java  … <dramatic pause > … a security disaster in a league of its own.

In a medium-sized organization (say, up to 10,000  PC endpoints) the problem seems to be even more painful due to a double whammy of already-high complexity and still-low resources. Java, Adobe Reader and  Flash, Firefox, Oracle fat clients as well as many vertical and business-specific applications are often patched MUCH later than Windows and Office. Quite a few of the organizations actually found success using their Microsoft tools for patching 3rd party applications. Microsoft WSUS and its older brother SCCM rule the roost there, but there are challenges with this approach as well (such as MSI package creation and all sorts of compatibility testing).

We have created a lot of great research on the subject of patch management as a part of overall configuration management (example). Similarly, patching as a method of remediating vulnerabilities is an obvious part of vulnerability management (example).  However, we still get calls about patch management planned and operated in isolation from either configuration management or vulnerability management (“no-no-no, we just need to know about patching!”).

One subject that presents a particular challenge is patching time frames (and, ultimately, SLAs). Across organizations and system types, they vary wildly -  literally from hours to years (!). A critical, known-to-be-exploited vulnerability in a DMZ Windows web server often gets fixed in 24-48 hours, while an internal database “medium” severity issue is likely to languish for years (exploited and re-exploited by literally generations of attackers lounging on your network).

These SLA are often affected dramatically by the extent of testing done to make sure that the patch does more good than harm. At this point, I’d venture a guess that testing is a major bottleneck – and one that can never be fully automated for fragile and complex server environments. Yes, you can have a nice staging environment, but not an exact clone of every system you plan to patch, complete with active users.  As a result, the users utilizing the patched system is the only real test, thus deploying patches to waves of users is popular as a way of ‘test-ployment’ i.e. testing combined with deployment. 

These challenges are, incredulously, leading some organizations to reduce the efforts they put in vulnerability scanning. After all, if every monthly scan gives you 10,000 findings, but you have enough bandwidth to fix only 300 (a real example), won’t you be at least a bit demotivated to scan? If you are going to assume you are owned, what’s one unpatched vulnerability between friends?  In any case, please treat this point as Anton thinking aloud and NOT as a recommendation to ditch you scanner. However, when you  do scan, you need to become MUCH smarter in picking what to fix first, second, third, etc …. as well as last and never.

Finally, please keep in mind that patching must have a purpose (darn, I feel stupid even saying this) – you patch to reduce a risk or to enhance application functionality. Patching for the sake of patching is just not a very useful mindset that also dulls this activity to an endless, mindless grind. Oh….wait

Possibly related posts:

• How Are We Doing Compared To Peers?
• Posts related to vulnerability management