Vulnerability Patching Like a Brain Surgeon

We’ve talked before about the approach of patching like a caveman. The logic behind that approach is that you can achieve at least 75% compliance with your patching policies with little effort. That’s 50% better than what many organizations achieve. But what do you do when 75% isn’t good enough? That’s when you bring in the brain surgeon.

Brain Surgeon Patching 

I spent about a decade specializing in patching. At the end of each patch Tuesday cycle, I usually had a clean Nessus scan. Any finding I had on that scan had a good reason attached to it, and a POAM (the DoD equivalent of a risk acceptance) to go with it. These were patches that were known to cause problems, whether they were Microsoft patches that caused systems to malfunction, or Java updates that our vendors hadn’t certified yet.

Yes, I was patching Java.

I’m not saying this is easy. One of our MSSP partners does both vulnerability management and patch management – both very successfully. His success rate makes his customers think patching is easy. It’s not. It’s a discipline… but I can relate. I left my last patching gig 12 years ago this month. I heard a story a few months after I left. Whoever they got to replace me wasn’t keeping up with my pace – and allegedly, my former boss commented to somebody that Dave didn’t have any trouble doing this. He thought patching was easy too, when I was sitting in a corner not talking to anybody and just following a process that took me years to put together.

I’m going to share my secrets – because they shouldn’t be secrets. I specialize in Windows, so I won’t be much help on the Linux or Macintosh side, but given the number of Windows systems out there, there are no shortage of people who can make use of this advice.

Just Reboot Already! 

Far and away the biggest problem I see in corporate environments is that they don’t reboot enough. They deploy the updates, their patching system calls it good, but the updates never take.

The reason for this is because Windows cannot replace a file when it is in use. So, if you deploy an update and don’t reboot, that file goes into a queue. Guess what happens if that file is still in the queue next month when you deploy next month’s updates? The new update is stuck behind the old one in the queue, and there’s no way for the new one to cut in line. if you fall behind on reboots, you can fall behind on patching. You’re leaving the money on the table.

This manifests itself in your vulnerability scans a couple of different ways. Two of the leading three vulnerability scanners have a finding related to pending reboots. In one of them, it’s labeled as a potential vulnerability. People love to ignore potential vulnerabilities. This finding is the reason you don’t want to do that.

The second way this manifests itself is when you see a string of Microsoft updates with the same title, except for the month and year. You can see this in any vulnerability scanner. When you see a system that is missing several of those updates, which all theoretically supersede one another, that is a good indication that those updates are waiting in line. If your patching tool says those updates have been deployed and your scanner disagrees, that’s what’s happening.

I once reduced a mega-corporation’s vulnerability count a bit over 50% just by providing a list of systems that needed to reboot. The longer the line is, the more reboots it can take to clear it. My usual advice is to reboot any system with a backlog at least twice a month until it catches up. If you want it fixed yesterday, reboot two or three times, scan, and repeat until the backlog goes away. You should see the backlog dropping.

ETL Files 

When you deploy a Windows update, Windows logs just about everything in a folder called C:WindowslogsWindowsUpdate. But it uses a weird file format called ETL. Microsoft provides PowerShell scripts to convert the ETL files into human readable format. The most useful one is simply called Get-WindowsUpdateLog, which tears through every ETL file on the system and creates an enormous plaintext file on your desktop. Have an experienced System Administrator open a PowerShell prompt, type Get-WindowsUpdateLog and hit enter, and wait for the file to generate. Open the file, search for the Microsoft KB associated with the patch that you are interested in, and look for errors.

What you have to do to fix any given problem requires experience, which is why I say to put an experienced System Administrator on this. While I can’t cover every detail, it’s news to many people that these logs even exist. Just knowing this resource is available to you is probably half the battle.

Forcing an Update 

There are times when a failed update will refuse to run again. You ran the update, it updated some files but not others, and the file that contains the actual vulnerability wasn’t one of the files that updated. But when you run the update, it may or may not give you the option to run the update again. Sometimes it just tells you that the update is already on the system.

When this happens, run the updater from a command line and include the parameter /? You may get lucky and find an option to run anyway. If the update doesn’t have that option, look for an option to extract the update. This will usually yield a file with a different extension but a similar size. If you don’t have that option, try extracting the update with Winzip or a similar utility. This extracted file doesn’t look to see if it’s already installed. It just skips the checks and installs.

A few years ago, I was working at one of the scanner vendors, of all places, and there was an update that had a ransomware scare associated with it. They deployed the update, but there were a few machines throughout the company that didn’t take, including mine. This was the trick I used to get my machine to update, and I passed the trick along to the CTO in case it helped any of my colleagues who were having the same problem. It let us get that emergency patch down in a hurry.

See, deploying updates doesn’t always work the first time even for security companies.

One More Thing… 

I want to add one more thing. When you deploy updates, there’s no reason to deploy old updates. If you have an Adobe reader vulnerability from 2020, don’t apply a patch from 2020. Apply the new one. There is no reason to do the work five times. Start with the newest updates and work backwards to maximize your effort.

But what about prioritization? Deploy the solution to your 10 biggest problems, according to our prioritization, and deploy this month’s updates, whatever they may be. Even if they are low severity. This month’s updates likely supersede some of the updates in your backlog, so deploying this month’s updates chips away at the backlog while stopping your backlog from growing.

Using this process, it’s possible to cut your risk score dramatically in a matter of months. Patching is a discipline. It takes discipline and knowing when to utilize your most experienced administrators. Keep in mind not every update necessarily deserves the brain surgeon approach. Critical vulnerabilities on sensitive systems that just refuse to apply deserve this approach. Medium or low vulnerabilities on test systems might not.

Knowing where to start can be difficult. Hopefully this helps my fellow security professionals to know what to ask for.

Patching is one of the few preventative measures available to us to secure our networks, and far too many professionals are reluctant to admit this is a problem. I hope that by pointing out the problem and pointing you towards solutions, you can help secure the networks that you’ve been entrusted to. Let’s work together to make this world a safer place.