← All posts · June 30, 2026 · By Brian Vitko

Shipping a firmware change you can’t roll back

Most of what a deploy tool does is forgiving. A package fails, you retry it. A script errors, you read the log and fix the script. A patch wedges a service, you roll it back. The whole job is built on the quiet assumption that a bad outcome is recoverable, because almost always it is.

The 2023 Secure Boot certificate rollout is not that.

Here’s the short version of the problem. The Secure Boot certificates Microsoft shipped in 2011 are aging out, and machines need the 2023 ones (the UEFI CA 2023 lineage) in their Secure Boot DB to keep trusting boot components going forward. KB5068202 is the mechanism. You stage the new certificate into the firmware’s allowed-signature database via a registry trigger, and it gets applied at the next reboot, down at the UEFI level, before Windows is even in the picture.

When it goes well, nobody notices. When it goes badly - incompatible firmware, a bootloader that ends up signed by something the updated DB no longer trusts, a vendor UEFI implementation that handles the DB write poorly - the machine does not boot. Not “boots slowly.” Not “boots to a recovery prompt you can script against.” It sits at a black screen, and you cannot RDP into a black screen. The fix is a desk visit, a USB key, and sometimes a reimage. The blast radius is physical.

So this is the rare deploy where the usual assumption is inverted. You have to assume a bad outcome is not recoverable, and build accordingly.

If you want it straight from the source, this is all Microsoft’s, not ours: the certificate-expiration and CA-update overview, the deeper Secure Boot key creation and management guidance, and the IT-pro Secure Boot playbook for the certificates expiring in 2026. We didn’t invent the deadline. We built the safe way to meet it.

The two ways tools get this wrong

I came up through hospital desktop engineering, roughly 30,000 Windows endpoints. I have watched fleet-wide changes go sideways, and I have a healthy fear of the ones that touch firmware.

When I looked at how you’d actually do this cert rollout at scale, I saw two failure shapes, and the tools fall into one or the other.

The first is to not touch it at all. Leave it to the admin. Which in practice means someone writes a script that sets the registry trigger, and then runs it across the fleet on a schedule, because that’s the tool they have and a schedule is what it does. The danger is invisible to the tooling. Nothing in the system knows this particular change can brick a box, so nothing in the system does anything about it.

The second is worse: treat it like any other patch. Push to a ring, push to all, watch the job status go green. The trouble is that “the job succeeded” tells you the registry write succeeded. It tells you nothing about whether the machine reboots afterward. A perfectly successful write of a value the firmware can’t survive looks exactly like a perfectly successful write of a value it can. The job is green either way. The machine is bricked or it isn’t, and the job has no idea which.

Both of these share one root mistake: they wait for a report of failure. And a bricked machine never files one.

Build around the thing that can’t report

That last sentence is the whole design. A bricked box does not send you an error. It goes quiet. So the alarm cannot be “wait for a device to tell you it failed.” The alarm has to be the silence itself.

Here’s what RFF actually does. None of this is aspirational; it’s the code that ships.

It’s off until you deliberately turn it on. The rollout is inert per workspace until an admin flips it on, and the toggle says, in as many words, confirm the KB5068202 value and pilot disposable VMs first. The dangerous button comes with a guard you have to consciously remove. You can absolutely choose to do this. You cannot do it by accident.

The pilot has to be disposable. Starting a rollout requires you to pick pilot machines, and the product tells you to use rebuildable boxes you can watch. Windows Server SKUs are excluded outright, and eligibility (a machine that hasn’t already rolled, and isn’t a server) is re-checked at the moment each ring fires, not just when you planned it. You don’t get to aim this at a domain controller.

It moves in rings, with a real soak. A pilot ring fires first. Then waves, sized by you. Ring N+1 does not fire until ring N has passed - and “passed” means it sat through a soak window (default four days) without anything going wrong. Not a token pause. Long enough for a machine to get rebooted by a human in the normal course of a workday, because the reboot is when the firmware change actually lands and the brick, if there’s going to be one, actually happens.

The canary is the silence. This is the part I’m proud of. Every few minutes, for every machine still in a soaking ring, RFF asks a simple question: this box was checking in when we fired the cert at it - is it still checking in? A machine that was alive at fire time, has since gone dark past the brick-silence threshold, and is not reporting the new certificate, is a brick suspect. And a single brick suspect halts the entire rollout immediately - it does not wait for the soak to finish, it does not fire the next ring. The absence of a heartbeat is treated as the failure, because for this specific change, absence is the most honest signal you’re going to get. A reported cert error halts it too, but we don’t depend on getting one.

The gate is the outcome, not the job. A ring only passes if the machines actually report the new cert in place and are still alive. “The deploy job returned success” is necessary and nowhere near sufficient. We learned this lesson the boring way on other features: a canary ring catches a failed action, never a successful action that did the wrong thing. (The same blind spot shaped our editable config baselines: removing a setting there means “stop managing it,” never “wipe it off every box.”) So the health signal here is the real-world state of the box, not the exit code of the script that nudged it.

When it halts, it halts loud, and the override is yours. A suspected brick stops everything and emails your admins. From there you investigate. Maybe it really is bricked and you just saved the rest of the fleet. Maybe the box is fine and someone just unplugged it for the weekend, and the silence was a coincidence. If it’s the latter, you can override the halt and let the rollout continue - because it’s your fleet and sometimes you know something the canary doesn’t. But you do it as a deliberate, audited click, with your name on it. We will tell you to stop. We won’t refuse to let you go.

What “we protect you as much as possible” actually means

It does not mean we made a firmware-level change safe. Nobody can. You cannot push a one-way UEFI change to real hardware and promise nothing breaks.

What you can do is make the dangerous thing impossible to rush, keep the blast radius tiny by default, and - the part everyone else skips - make the alarm fire on the one signal a bricked machine actually emits, which is nothing at all. Off by default. Disposable pilots. Rings that won’t advance on a silent box. A gate keyed to whether the machine lives, not whether the job claimed victory. And a halt that respects you enough to let you override it on purpose.

This is exactly the feature I wanted at my old job and never had: a way to do the genuinely scary fleet operation where the tool is on your side about the danger instead of pretending it isn’t there. You still have to decide to roll the certs. RFF just makes sure that if it starts going wrong, you find out from the second machine, not the two-hundredth.

Secure Boot cert rollout is part of RFF, free for your first 100 endpoints. Turn it on when you mean it, pilot disposable boxes, and let the canary watch the rest - getrff.com.