Top 10 patching hurdles and how to overcome them
Patching is the most effective, efficient and simple method to mitigate malware, worms and viruses. It may not protect against advance attacks that make use of 0-day vulnerabilities, but for the most part it is an excellent cost effective and reliable solution. But many organizations have a hard time patching. This made me want to examine: what challenges exist and how can we address them? Here are some common hurdles for patching and tips that organizations can use to move toward a better patching posture.
1. Unknown assets
Before patching, one needs to find what assets are affected and therefore eligible for a patch. Larger corporations spread across countries or continents have a hard time getting an accurate handle on the presence of their assets. And if an asset is not known, it cannot be patched.
Recommendation: Use an asset management tool, inventory control system or a similar process to monitor assets. No tool is perfect, so try out different ones and select what suits your needs. Usually a combination of multiple approaches gives the best results.
2. Many patches require assets to betaken offline
Mission critical systems cannot be taken offline, but many patches require a reboot, which would result in a downtime.
Recommendation: There are some highly available products as well as many operational tricks that can be experimented with to avoid downtime. The solutions are different depending on the software in question — a chat with your operation folks or system administrators can give you some ideas to begin. Use your solution in a test environment first before deployment, and if possible group down times together.
3. Scarce IT resources
It is safe to say that most organizations lack sufficient IT staff to cover all needs. IT is always stretched thin and many times resources are not available for quick deployment of all patches on the world wide assets.
Recommendation: While most IT departments use some sort of a patch management system, many of these systems are excellent in one area (like Windows patches) but weak in others (like database patches). Use a combination of manual and automated approaches to cover your entire asset base. Properly managed networks and assets go a longway when it comes to patching.
4. Unreasonably long patch test cycle
Before patching, it’s logical to test if the patch will not break anything. For example it make ssense to test if a critical home grown custom application works correctly after the patch is installed. While a good practice, it can sometimes can buried in bureaucracy and then takes too long, giving attackers valuable time to create worms that target the unpatched machines.
Recommendation: Start with prioritization of assets and the applicable patches. Consult developers, testers and other system administrators for their opinions. You will be amazed how their input can cut down test cycles because incompatibilities are caught early on.
5. Too much virtual patching or workarounds
Virtual patching is a concept where instead of installing the actual patch organizations deploy a firewall rule, an IDS signature or an antivirus update that prevent attacks exploiting the vulnerability that is fixed in the patch. This technique has its merits and can be very useful to buy valuable time, but relying on it for too long can land you into trouble, because you have to track which assets have what virtual patches.
Recommendation: Whenever possible apply the real patch from the vendor. Use virtual patching only as a temporary arrangement while the patch is being tested or software is being modified to work with the newly released patch. Also do your homework — make sure you benefit from virtual patching where you get the best returns (for example in web applications), which is a bit of a different area than traditional virtual patches.
6. Conflicting binaries
Sometimes a patch from Vendor A may not install successfully due to binaries installed by a patch from Vendor B. Over the course of year I think this has improved but once in a while you will face this situation.
Recommendation: If possible, do not overload the same server with products from multiple vendors and use dedicated servers for discrete business function to reduce conflict between multiple software programs. A good infrastructure to test will go a long way in identifying these conflicts early.
7. Third party patches
While your vendor is working on developing a patch to fix a vulnerability, the security researcher (or his company) that identified the vulnerability released his or her private patch. Do you install this third party patch or wait for a patch from the vendor?
Recommendation: The urgency of the situation and credibility of the third party patch creator play a vital role here. For most situations I advise not installing such a third party patch as it may break something else, may not be correctly tested or in the worst case could come from an imposter with malware embedded inside the fake patch. Try to implement the workaround provided by the vendor first before exploring third party patches.
8. Expired Licenses
What do you do when a new patch will not download or refuses to install because the software license or support has expired? Mistakenly expired licenses or deliberately used pirated software will not install patches.
Recommendation: Mistakenly expired licenses mostly reflect lapse in the administration of the system. Try to find the root cause and make sure you have proper policies in place. In some countries pirated software is common, but creates a breeding heaven for viruses and worms. Make sure you tie up with respectable vendors. Use vulnerability scanners or asset managers to finding unauthorized installations of software in your organization.
9. Patching kiosks, industrial control systems or critical infrastructure
Kiosks, industrial control systems, critical infrastructure or SCADA systems are increasingly using standard operating systems like Windows or Linux. But their version of Windows may be a slightly modified version of the standard OS – which has been customized by the SCADA vendor, so when Microsoft releases a patch, it cannot be installed on these systems. These systems have to wait for the customized or recompiled patch from the SCADA vendor.
Recommendation: Demand from your vendor expedite re-release of such custom patches. If the standard patch can be installed; demand from your SCADA vendor some sort of guidance on the safety of installing the standard patch on your critical infrastructure or factory floors.
10. Large number of patches
In the last few years we have seen unusually high numbers of patches and patches which fix tons of vulnerabilities. It’s very difficult to cater to such a high number of patches.
Recommendation: We cannot control how many patches are released by vendors. But with proper asset management, patch management tools and correctly maintained infrastructure we can prepare ourselves better for the dreaded patch day.