A green job and no software: what winget’s exit 0 doesn’t tell you

Here is a job log from my own production tenant, two days ago:

winget --scope machine failed for Postman.Postman (0x8A150010);
retrying without scope constraint...
SUCCESS - winget 0x00000000

Green job. Every indicator I had said the deploy worked. Postman was not on the machine.

This is the worst class of bug an RMM can have. A failed deploy is annoying and you fix it. A deploy that reports success and does nothing is a lie you build on top of - you close the ticket, you mark the fleet compliant, and six weeks later somebody asks why the thing you rolled out in June isn’t anywhere.

The root cause is a Windows detail worth knowing whether or not you ever use RFF.

0x8A150010 is not a soft failure

When you install software fleet-wide you install it machine-wide. Per-user installs land in one profile, which on a shared machine is the wrong profile roughly as often as it’s the right one. So the first thing our agent tries is winget install --scope machine.

Sometimes that returns 0x8A150010. Most tooling treats a non-zero exit as “didn’t work, try something else”, and that’s what we did: retry without the scope constraint.

That’s wrong, and specifically it’s wrong in a way that produces a green job. 0x8A150010 means winget looked at the manifest and there is no machine-scope installer for this package. The package is per-user only. It isn’t a transient failure, a permissions problem, or something a retry can fix. It’s a conclusive answer to the question “can this go machine-wide”, and the answer is no.

Retry it without the scope constraint and you get one of two outcomes, both bad:

It appears to work. The agent runs as SYSTEM, so a per-user install with no scope constraint installs into the SYSTEM account’s profile. C:\Windows\System32\config\systemprofile. winget exits 0 and it is being completely honest, because it did install the software. Into a profile no human will ever log into.

It hangs. Discord did this for over 900 seconds in our lab. We have one logged at 24 minutes.

Our code made this worse in a small, specific way. It overwrote the rung-1 exit code with the retry’s exit code, and only afterwards checked whether the result looked like a user-scope problem - by which point the 0x8A150010 that would have told us exactly what was going on had been thrown away.

The fix is four lines: keep the machine-scope code in its own variable, and short-circuit 0x8A150010 straight to the fallback that borrows the signed-in user’s token. It’s also considerably faster, because it skips a retry that was never going to succeed.

The one that surprised me: winget can’t unpack a zip as SYSTEM

While fixing that I found a second code doing the same damage: 0x8A15005C, archive extraction failed.

Four packages failed identically on a real endpoint, all with the same log signature:

Successfully verified installer hash
Extracting archive...
Failed to extract the contents of the archive

MobaXterm, Paint.NET, Process Explorer, TeamViewer. My first assumption was catalog rot - bad manifests, moved URLs, the usual. So I A/B tested on the same box: same four packages, same winget, same machine, running under a normal interactive admin token instead of SYSTEM.

All four installed cleanly.

It isn’t the packages. winget’s archive extraction does not work when running as SYSTEM. That’s a context limitation, and it’s the kind of thing you can only learn by testing in the context your software actually runs in. I’d have happily filed four “broken package” bugs and been wrong about all four.

Between the two exit codes, six catalog packages were fixed outright: Postman, Discord, MobaXterm, Paint.NET, Process Explorer, TeamViewer.

Why I didn’t find this sooner

Because I was testing the wrong thing.

We had a catalog validator. It checks that all 131 package recipes are well-formed and that their download sources still resolve. That’s genuinely useful and it catches real breakage. It also cannot tell you whether an install lands, and it definitely can’t tell you whether it lands somewhere a human being can see it.

So the fix was a second harness that installs the software for real, and the design constraint was the entire point:

It runs as SYSTEM. Not over PowerShell Direct’s admin token, not as an interactive administrator. It registers a scheduled task running as SYSTEM, because the agent is a SYSTEM service and an install validated as an interactive admin proves something no customer ever runs. That distinction is the whole reason both bugs existed.

It replicates the agent’s ladder rung for rung - --scope machine, then the no-scope retry, then the ARP check - rather than testing an idealized install path. Validating a code path nobody runs proves nothing.

It checks visibility, not just exit codes. After each install it diffs Add/Remove Programs across both registry views and every real user hive, with S-1-5-18, S-1-5-19 and S-1-5-20 excluded. Counting the SYSTEM profile as “visible” would defeat the entire check, which is precisely the trap the Postman bug fell into.

It reports six verdicts rather than pass/fail: OK-MACHINE, OK-NOSCOPE, OK-NO-ARP (installed, but writes no ARP entry, so it’s invisible to inventory - Blender is the classic), USER-SCOPE-ONLY, NOT-FOUND, FAILED. “It worked” and “it worked and you can see it” are different answers and deserve different words.

It also refuses to run anywhere but one disposable VM, matched on both the host name and the guest computer name. It installs 120+ pieces of third-party software in a loop. It should be structurally incapable of doing that on anything I care about.

The harness had the same bug I was hunting

My favourite part. The first version of the harness reported timings that made no sense: Google Chrome at 1975 seconds, PowerToys at 898.

Neither installer is slow. Process.WaitForExit() returned true, and I believed it. But WaitForExit() returning true does not mean the output pipe is closed - a grandchild process holding the stdout handle keeps it open long after the installer you launched has exited. I was timing my own harness draining a pipe, not an installation.

Then batch 9 returned 4 results out of 10 packages. An installer rebooted the guest mid-run, the worker died, and the summary printed a clean pass over the six that never ran at all. A missing result read as a good result.

Both are the same failure the harness was built to catch: something reported success because nothing checked the specific thing that mattered. There’s now an explicit UNTESTED record when the worker stops early, and a bounded drain on the output pipe. If you build a tool to catch lies, budget time for the tool lying to you.

What’s still wrong, on purpose

The ARP phantom check still asks “did any new app appear” rather than “did this package appear”. Any unrelated software change on the box during the window satisfies it. I know it’s weak.

I left it alone anyway. Making it package-specific means matching install names to ARP display names across 131 recipes, and getting that mapping wrong sends genuinely-successful installs down the user-session fallback path - trading a rare false pass for a common false failure. The exit-code short-circuit already removes the case that actually bit us in production. The stricter check deserves its own change, with its own evidence, on a day when it’s the thing I’m working on.

The part that generalizes

If you take one thing from this, make it this:

An exit code tells you a process finished. It does not tell you the thing you wanted to happen happened.

Everything downstream of that gap - your compliance report, your deployment dashboard, your green checkmark - inherits it. If you deploy software as SYSTEM, and nearly every management tool does, then “installed” needs to mean “present in a place a signed-in user can actually reach”, and something has to verify that independently of what the installer claimed on its way out.

For us that meant admitting the validator we had was checking the easy half, and building the expensive half. Six packages were quietly broken in the catalog for weeks. They were broken in production, on my own machines, in a product I use every day, and every job that deployed them came back green.

Both fixes shipped in agent version 1.3.15. Free-plan agents pick updates up automatically; if you control your own rollout, anything at or above 1.3.15 has the fix. The exit-code handling covers 0x8A150010 (no machine-scope installer) and 0x8A15005C (archive extraction under SYSTEM); both now route straight to the signed-in-user fallback instead of a retry that can’t succeed.