Patch management is not just IT’s responsibility, get your whole team on board

I have been on the road for a few weeks now and surprisingly the topic of discussion has predominantly been patch management. Why is patch such a prevalent topic?

Patching responsibility

As we near Patch Tuesday this month I wanted to share a few thoughts and observations with you all from my recent travels. Recently I attended RiskSec in New York, and last week I was at Infosecurity Europe.

At RiskSec I had the pleasure of participating on a panel discussion about patch management. As I was preparing for the discussion I was about to have with Charles Kao (SVP, Cyber Security at EthicalHat) and Sandy Bilus (Partner, Saul Ewing Arnstein & Lehr), one of the editors from SC Media joined us. He had one question, which was, “Why are we STILL talking about patching?” We went on to discuss this question for a bit and he said that he knew full well why. Because it was still a topic that many companies struggle with even now. So, let’s start there. Why is patch such a struggle?

Patching is often the responsibility of the operations or infrastructure team. They are required to keep systems up to date, but rarely have the full authority to do so. See if any of these situations ring a bell:

• While negotiating maintenance windows for the server infrastructure, an application\business line owner informs you that their system is critical to the business and cannot be brought down for any reason including maintenance.
• A high-ranking exec chews out your boss because his system rebooted at a less-than-convenient time. It does not matter that he/she was prompted to reboot for the past 24 hours and ignored it, and the system finally forced the event. Never the less, said exec is now placed in an exceptions group and has a vulnerable machine going forward.

There are many other examples of how little power some admins have when it comes to executing their assigned tasks, but you get the idea. So, how to deal with this? Well at some point you need to grab the bull by the horns and face the problem head on. Here are some tips.

Get the right people involved

Patching is everyone’s responsibility whether they realize it or not. A successful patch program includes executive buy-in (not exceptions) and involves partnering with the business (Those critical apps really need to be patched. Remember Struts, anyone?).

It’s vital to include the users (they know how to test the applications they use better than IT does). Everyone is responsible for patching, whether they are working with the team to find a convenient window to patch critical systems, or backing the team when there is resistance to the patching policy. Or maybe you have a user who needs to save their work once a month and actually closes all apps and reboots their system.

Exceptions are sometimes necessary, but should not become the norm

If there is a need for an exception at a system level or a patch level, that exception should include additional info, not just be entered as an exception. If there is a system that cannot be patched, there must be a clearly defined reason and the person who enters the exception should be the person who is demanding the exception (aka the business line owner, app owner, etc.) This forces accountability on the people responsible for blocking the process, not passing that accountability on to the person whose hands are tied. There should also be an approval process executed by a cross-functional team including management, security, and operations.

If a specific patch has a known issue it should be documented, but should also include additional mitigation like segregating the vulnerable system, restricting access, running additional security controls on the system or enforcing more strict controls. The exception should not be left as a permanent exception either. Is the issue dependent on an upgrade from the vendor, a bugfix (likely in an upcoming release), etc.?

Communication is key

I hear this one a lot – “Oh, they patched last night. It is their fault that this system is broken” – when in fact that system wasn’t touched. A lot of times people blame patching for any problem that happens close enough to the event. Communication is key to ensure people are aware of what is being patched, when it is being patched, and confirmation that patching was successful.

At Ivanti I receive several communications regarding patching each month. I get the general patch maintenance notification that all of our corporate users receive. This informs me of when patching will occur and includes contact information in case there is an issue, as well as details about what is required of me, the user, in this window (reboot in this timeframe or it will force you to do so within xx hours).

As a stakeholder in many of our cloud and customer-facing services I also get informed by the DevOps and CloudOps teams when critical systems are being patched. Similar information is included on what is being patched, timeframes, etc. I also receive a confirmation when patching is resolved for these critical systems. This way I am aware that if something happens during the window it is likely being executed on already, so I can reach out to the team performing the update first to confirm they are aware. If it is post-patch window and the system was one of those being updated, I know to escalate the issue quickly as something could have potentially been affected by the updates.

I could go on with other best practices. As I said before I am on my way to Madrid for our corporate event and one of the presentations I am giving will be on patch management best practices. Every year I end up running this session multiple times because it will get overbooked.

Patch Tuesday forecast

Our monthly trend of zero-day exploits continues this month with yet another attack targeting Adobe Flash. Adobe released a Flash update with the security details provided in APSB18-19. This update addresses four vulnerabilities—two rated Critical and two rated Important. According to various sources it is CVE-2018-5002, a stack-based buffer overflow bug, which is being exploited. This vulnerability requires the system user to open a document containing a weaponized Flash Player object. The buffer overflow is exploited as the flash object opens and additional shell code is downloaded to the system for deeper exploitation.

Forecast for June:

• Adobe released their Flash Player update, so we know what Microsoft will include it. No additional Adobe releases are expected, but you never can tell.
• Mozilla released a security update for Firefox 60.0.2, Firefox ESR 52.8.1, Firefox ESR 60.0.2 on Wednesday under their Security Advisory 2018-14.

Having these security releases out the door a week early, nothing major is expected from other third-party vendors and no major surprises are anticipated from Microsoft. This could be a relatively calm Patch Tuesday. But then again. there are those new Spectre and Meltdown vulnerabilities we reported last month.