ENGINEERING

From USB stick to enrolled machine, hands off

The last mile of Windows deployment is still absurd. You image a machine, then you go find it again to install the management agent, then you go find it a third time because the agent installed but never enrolled. Three touches for one box, and the second two happen days apart when you have forgotten which machines you already did.

What we wanted was one touch. Plug a stick into bare metal, walk away, and have the machine appear in the console named, enrolled, and waiting for approval.

That now works. The interesting part isn’t that it works, it’s where the bugs turned out to be hiding.

Every one of them got past unit tests, past build-time checks that refuse to ship malformed media, and past more than one careful read of the code. Not because those checks were weak, but because none of these bugs existed yet. They only came into being when a real machine booted: in the handoff between a batch interpreter and PowerShell, between a console input buffer and a prompt, between what a firmware reports about itself and which branch of an XML file that sends you down. Every layer involved was behaving exactly as documented. The defect was in the seam.

And every one of them was obvious within ninety seconds of writing the stick and booting it.

The chain

Boot the stick. WinPE comes up and asks one question: what should this machine be called. Type a name or press Enter to skip. Windows is applied, the machine renames itself during specialize, OOBE runs silently, a scheduled task fires on first boot, the agent installs and enrolls with a token baked into the media, and the device lands in the console as pending approval.

Pending, not active. A stick that enrolls machines silently is a stick that can enroll a machine you didn’t intend, so the token is per-stick, revocable on its own, and an operator has to approve the device before it can be dispatched a single job.

The whole thing is OSDCloud underneath, plus an unattend file, plus a scheduled task, plus a token. None of that’s novel. What follows is.

Bug one: yesterday’s ISO with today’s label

The first field failure was reported like this: “I named it WKS-042 and the logs say wks-041.”

The build script rebuilt the ISO, wrote a fresh stamp file recording the name and timestamp, and handed over the media. The ISO build had failed. The old ISO was still sitting there from the previous run, so the script cheerfully delivered yesterday’s media with today’s label.

The stamp said WKS-042. The bytes said wks-041. Both were internally consistent. Nothing errored.

The fix is the boring, correct one: delete the target first, verify it exists afterward, and only then write the stamp. If the build fails you now get nothing, loudly, instead of something wrong, quietly. The guard fired for real a few hours later when a VM still had the old ISO mounted in its DVD drive.

Bug two: a percent sign

WinPE runs startnet.cmd, a batch file. We build a PowerShell one-liner into it, including a countdown:

if ($i % 10 -eq 0) { Write-Host "rebooting in $i seconds" }

cmd.exe expands % before it processes quotes. It saw % 10 -eq 0) { Write-Host "rebooting in $i seconds" } , found no matching variable, and deleted the lot. The resulting line was a parse error. WinPE dropped to X:\ and deployed nothing at all.

A percent sign. In a comment-free, perfectly valid PowerShell expression, inside quotes, that never reached PowerShell.

The countdown now computes its marks on the build machine and emits a %-free string. There’s also a build-time check that refuses to write a startnet.cmd containing a percent sign, because we’re not going to remember this in six months.

Bug three: the quotes came off too

Same line, different layer. This:

Write-Host ("  rebooting in $i seconds ...")

cmd strips the inner double quotes out of a batch-hosted powershell -Command "...". Every tick of the countdown threw CommandNotFoundException. Non-fatal, so the machine still rebooted and still deployed, but the console spat an exception sixty times on the way out.

It only appeared once bug two was fixed and the countdown ran for the first time. Fixing one bug is how you meet the next one.

Bug four: the keystrokes you already typed

The name prompt is a Read-Host. In the lab we were spamming Enter to get past WinPE’s “press any key to boot” window.

Those keystrokes don’t evaporate. They queue in the console input buffer, and Read-Host eats one the instant it appears. The prompt was answered, correctly and instantly, by a keypress from ninety seconds earlier. The machine deployed unnamed, and the screenshot of the prompt looked completely normal because the answer arrived faster than the screen could show the question.

Two fixes: flush the input buffer immediately before prompting, and require a second Enter to confirm you really meant to skip naming the machine.

Bug five: the name that reached the wrong branch

There are two paths through the unattend file. When the machine has a TPM, OSDCloud writes its own specialize section and we append to it. When it doesn’t, we create the section ourselves.

Every VM took the create branch, because the TPM reports “Not Supported”. Every piece of real hardware took the append branch. The computer name was only being written on the create branch.

So: worked perfectly in the lab, silently dropped the name on every physical machine. The tell in the field was a device that showed up called desktop-7g4kq2p - the random name Windows assigns when nobody tells it otherwise.

This is the one that best explains why booting matters. A VM isn’t a smaller version of a laptop, it is a different code path, and the difference was one attribute in a firmware report.

Bug six: a break-glass account nobody could use

If enrollment fails, you need a way into the machine. There was a local admin account with a random password, and that password existed in exactly one place: a base64 blob inside an unattend file on a disk you couldn’t log into.

That isn’t a recovery path, that’s forensics.

The password is now written in plaintext to a file on the machine’s own disk, readable from any install media. That sounds alarming for about five seconds, until you notice that physical access to an unencrypted disk already grants a local password reset. It gives up nothing that a screwdriver did not already give up, and it turns a bricked enrollment into a two-minute recovery.

That was the interim answer. The permanent one is escrow, and it now works like this: the agent reads that file, sends the password to the console encrypted, and deletes the file from the disk only after the server confirms it holds it. Every other outcome - a 404, a 500, a captive portal, a server that hasn’t deployed yet - leaves the file exactly where it is and retries later.

That ordering is the entire design. Deleting on anything less than a confirmed escrow would destroy the only way into a machine whose agent may itself be the broken thing.

Centrally, the password is DataProtection ciphertext under its own purpose, written only by the agent channel and never by any user-facing endpoint, and revealing it’s admin-only and writes an audit row in the same transaction as the read - so no reveal can happen without a record of who did it. That’s not a new design; it’s the same custody the BitLocker recovery keys already use, which is a strictly more dangerous secret.

And until the delete is confirmed, the device page’s row reads “also on the device’s disk”. A tool that quietly implies a secret has been cleaned up when it has not is worse than one that never claimed to clean it up at all.

What I would take away from this

Six bugs. Four of them produced no error at all - a wrong name, a silent no-op, an unnamed machine, an unusable account. The two that did error still let the deploy finish.

None of them were logic errors in the sense a reviewer catches. They were interactions between layers: batch and PowerShell, console buffers and prompts, firmware reports and XML branches, build scripts and the filesystem. Every one of them lived exactly where two systems met, and each system was individually behaving correctly.

That class of bug doesn’t exist in your code. It exists in the gap between your code and something else’s, which is why a test that stubs out the something-else will pass forever. The one thing that reliably surfaces them is running the real artifact on real hardware and watching what happens.

So the checks moved to where the bugs actually were. The build now refuses to write a startnet.cmd containing a percent sign, or an unbalanced quote, or a control character, or anything that fails to parse as PowerShell - and it verifies the ISO it just claimed to build actually exists before stamping it. Those tripwires exist because of the specific mornings that produced them.

The imaging chain is proven now, end to end, on both a VM and physical hardware: typed name survives, rename happens, OOBE is silent, the agent enrolls, the device lands pending, approval releases it, and the scheduled task deletes itself. Ten to fifteen minutes, one touch.

The three touches are down to one. It took six bugs that only a booting machine could have shown us, and six tripwires so that no one has to find them twice.