How to Approach Security Patch Management

If you ask a system administrator, they would probably say security professionals underestimate how hard it is to patch vulnerabilities. As a former system administrator who specialized in patching before moving to vulnerability management, I understand exactly why this is true.

Not all security patches are created equal. For that matter, not every instance of the same patch is created equal.

Generally speaking, there are two styles of patching, and neither of them is appropriate all the time. If you’re unhappy with how quickly your enterprise patches, or with its success rate, this has something to do with it. I fixed somewhere around 800,000 vulnerabilities during my sysadmin career. That wasn’t my team, that was me. I was a team of one. The only reason I was able to do this was because I blended the two approaches. The tooling was secondary. I could get the updates down successfully regardless of the tools my then-employer let me use, though some did make life easier for me than others.

Let’s talk about the two approaches.

The Caveman Approach to Patch Management

The first phase of my patching was dumb, lazy, and passive. It always gets people’s attention when I say I don’t want you to work smarter or harder, I want you to work dumber and lazier. But you can get somewhere between 70 and 90 percent of your patching done by just deploying all the patches your patch management system thinks are missing, suppressing the reboot, then going to lunch. Then, during your approved maintenance window, reboot the systems you patched. If you have time, reboot them a second time.

If a 70 percent success rate is good enough, that’s probably all you have to do.

If a 70 percent success rate sounds like an improvement over what you’re getting now, you don’t have to work smarter. Work dumber and lazier. That’s probably not good enough, if only because you can’t choose which 70 percent. Any time you don’t get 100 percent, someone will ask how you know you fixed the right ones. I speak from experience. That’s what Nucleus is for. More on that in a minute.

But getting at least 70 percent of the way to where you want to be with minimal effort saves you energy that you need to get the rest of the way. Patch management is not a sprint. It’s not even a marathon. It’s a series of marathons in sub-optimal conditions. I’m dating myself with this reference, but when I was a kid, there was a Canadian athlete named Terry Fox who lost his leg to cancer at the age of 19. He decided to run a series of marathons across Canada on a crude artificial leg to raise money for cancer awareness. He ran a marathon a day for 143 days until his health wouldn’t allow him to continue.

That’s the best metaphor I’ve found for patch management. Running a marathon a day across Canada is difficult even for a skilled distance runner in peak health. Most sysadmins tasked with patching are doing a very difficult task and they aren’t doing it under optimal conditions.

Grit and determination wasn’t enough for Terry Fox to do what he did. It was certainly part of it, and he certainly had a superhuman level of both. But the fundamental reason why Terry Fox could run marathons and I can’t is because Terry Fox knew how to pace himself. His tremendous skill at pacing himself enabled him to keep going even as the effects of injury and cancer piled up.

We can apply the lesson of Terry Fox to IT and infosec. Getting as much done as you can with as little effort as you can was one of my key secrets to successful patch management when I was a system administrator.

The Brain Surgeon Approach to Patch Management

Once I reached the limits of what my dumb and lazy approach to patching could get me, I shifted gears. I am (or at least was) a skilled systems administrator, so I was capable of investigating why a patch failed to install and taking corrective action. These frequently required review on a case-by-case basis and could become very time consuming. Each percentage point above 90 percent can take as much effort to achieve as the first 90 percent took in total.

The first question is whether the update is worth the effort. That’s what Nucleus is for. I would argue that on a low severity vulnerability, a 70 percent success rate is good enough. It’s the criticals and highs that deserve the heroic efforts required to get to 100 percent.

Troubleshooting a patch that just doesn’t want to install is a very underrated and under appreciated skill. Typically this involves manually installing updates and looking for errors, examining log entries, manually uninstalling and reinstalling, passing little-known command line switches to the update, forcing file updates via WMI, multiple reboots, and in extreme cases, support cases with vendors. The skill required and the level of effort is similar to conducting a forensics investigation on a system. And based on conversations I’ve been involved in, I don’t think many security professionals know that. One of the best things we can do as security professionals is to understand the work required to complete an effort, and adjust our expectations accordingly.

When I was a sysadmin, I handled both the caveman phase and the brain surgeon phase. But you can split up the work among multiple people if you have multiple people available. When I was doing vulnerability management at a Fortune 20 company, we had three people rotating in and out of the two phases as needed. More would have been better, if the team had been large enough.

The methodology served them well, and it’s worked everywhere else I’ve had the opportunity to try it. In the most extreme case, a team I worked with fixed 1.3 million vulnerabilities in one epic summer.

Vulnerability Tech Debt is Like Personal Debt

When a customer was faced with a large backlog, I’ve also used the model of debt. Not everyone agrees that every missing patch qualifies as tech debt, but the model works. Hear me out.

I’m one of those people who paid off all my personal debt in less than 7 years. The formula to do that isn’t super complicated. Pay this month’s bills as they come in, and take the recurring bill that hurts the most and pay extra on it. Then when you pay one bill off, you take what you were paying on that bill and apply it to your next one. The amount you pay every month stays consistent, but over time you’re paying fewer and fewer creditors.

You don’t have to be a financial genius to do it, and you don’t have to be especially wealthy either. It’s really about making a budget and a plan and sticking with it every month. You can take a similar approach to patching. Push out this month’s updates as they come out. That stops your tech debt from growing. Then pick 10 updates from your backlog, ideally, 10 updates that are hurting you the most, and deploy those as well. Use the caveman/brain surgeon tandem on those updates to bring your tech debt under control.

Getting patching under control requires a plan, and it requires sticking with it every month. But once you build it into your processes and it becomes part of your culture, there’s no reason why it has to feel like an impossible dream. And counter-intuitively, it starts with learning when to work dumber and lazier.

Because when I’ve studied vulnerability management programs, frequently the underlying problem I found was that the system administrators were taking the brain surgeon approach all the time. And that meant they didn’t have enough time or energy left to finish the work.