Letting an AI live-fire test my RMM: 118 tests, 25 findings, and the guardrails that kept it sane
At about 1am, while I was asleep, an AI agent running my test plan discovered that a successful agentless RemoteRun was destroying the target device’s inventory. The job log said “Inventory updated.” The device went from 74 populated inventory fields to 2. No error, no alert, nothing. The feature worked, reported success, and quietly wrecked the data underneath it, and it was worst on exactly the agentless boxes the feature exists for.
I didn’t find that bug. I was unconscious. The agent found it, filed it with a reproduction, fixed it in the endpoint agent, republished, re-ran the same operation, and by morning the report read: dc1 inventory 68 fields before, 68 after, regression checked, local Run still refreshes correctly.
That’s the pitch for agentic testing in one paragraph. The rest of this post is the part vendors skip: what it got wrong, and the guardrails that made it safe to let an AI push buttons on my product all night.
The setup, honestly
Nothing about this is elegant. The agent doing the driving happens to be Claude Code, but nothing in this story depends on which agent you use; the interesting question is whether an AI can operate a real test environment overnight, investigate failures, modify its own harness, and validate fixes without being allowed to hurt production.
There’s a folder called rff-testdash with seventeen generations of a Python test
runner in it, because the agent iterates on its own harness the way you’d expect:
runner.py, runner2.py, runner14b.py. Alongside those sit
Playwright scripts driving real Chromium against the web app, SignalR clients probing
the live hubs, and Stripe checkout flows running against test mode.
The pieces that matter:
- Everything targets a staging clone. Separate API, separate web app, separate database, separate object storage, Stripe in test mode. The production system never sees a test.
- Real Windows underneath. A lab domain with a DC and member machines, because an RMM tested against mocks is an RMM tested against fiction. Hyper-V checkpoints get taken before each run.
- A live dashboard. A tiny threaded HTTP server serves status.json and the findings log so I can watch from bed or check the score with coffee. 118 tests on night one: 104 passed, 6 were the harness’s own bugs, 8 deliberately not run.
- A written report waiting in the morning, leading with what broke, what got fixed, what needs a human decision, and what state the lab was left in.
What two nights actually produced
The exact accounting, so “finding” is never doing ambiguous work:
Night one’s run:
| Result | Count |
|---|---|
| Tests executed | 118 |
| Passed | 104 |
| Harness’s own failures | 6 |
| Deliberately not run, handed to a human | 8 |
| Apparent failures investigated | 24 |
| …that were real product defects | 5 |
| …that were the harness’s own bugs | 19 |
The findings log, both nights:
| Result | Count |
|---|---|
| Numbered findings written up | 25 |
| Confirmed defects (product, structural, or billing) | 21 |
| Fixed and re-validated before I woke up | 11 |
| Claims the agent retracted as its own error | 2 |
| Suspected, deliberately left unreproduced | 2 |
A sample, so you can judge the class of bug this catches:
- The shipped “Domain Controllers” smart group had been empty forever. Every domain and AD group rule compared against a field that was never populated, silently matched nothing, and a job aimed at an empty group dispatches to zero hosts while reporting success.
- A read-scoped API key could acknowledge performance alerts. Read scope, write effect.
- The vulnerability dashboard read 0 critical, 0 high, because severity strings were compared case-sensitively.
- Paging on the findings view duplicated about 85 rows and made a different 85 unreachable, the kind of bug you only see with real data volume.
- A job’s timeout was only enforced for agents that had stopped heartbeating. A wedged payload on a healthy agent ran forever and pinned a concurrency slot.
- Job timeouts, uninstall ladders, reboot-resume, approval gates, cross-tenant isolation (8 for 8), and hub authorization (10 for 10, no leaks) all held. Verified working is a finding too.
It also ran the billing gauntlet with test cards and caught Stripe charging a price the product no longer advertises. Software bugs and money bugs fall out of the same process.
The number nobody puts in the demo
Across the first run, 24 apparent failures produced 5 real defects. The other 19 were the harness’s own bugs: wrong assumptions, wrong timing, wrong parsing. The agent also published one confident root-cause analysis it later had to retract, and one overclaim that happened because it generalized from the single machine whose inventory it had itself destroyed an hour earlier.
Every retraction is in the log with a rule attached. The headline rule the run converged on: when a test fails, the prior probability says the test is wrong. Prove the test correct before reporting the bug. That rule is now part of the standing instructions, and the false-alarm rate dropped hard on night two.
If someone sells you autonomous AI testing without an error-rate section, they have an error-rate section, they just haven’t read it.
The guardrails that made this sane
None of this would be survivable pointed at production with my credentials. The setup works because the blast radius is designed before the intelligence is applied.
Blast radius first. Staging clone, test-mode billing, lab VMs with checkpoints taken before anything runs. The worst possible night ends with a checkpoint restore.
The AI has its own identity. It authenticates with its own named API key, so every mutation it makes lands in the product’s audit log attributed to that key, the same audit trail customers get. When it created 54 jobs, the log listed every one by name. Misbehavior means revoking one key, not changing my password.
Two gates, and the human one counts. The agent validates its own fixes on staging, with evidence: before-state, operation, after-state. That’s gate one. Gate two is me, manually, before anything promotes to production. The agent can push to the staging branch all night; only I promote. Its own report ends by saying the manual gate is the one that matters, and it’s right.
A refusal list. Some tests were deliberately not run: real Windows Update passes (hours long, installs real patches, reboots), anything that kills the session the other tests depend on, anything irreversible without a human watching. The agent’s job on those is to stop and say “this needs you,” and it did.
The product’s own safety gates apply to the AI too. Approval-gated jobs blocked the agent’s dispatches until approved, the same way they block a compromised operator account. Row-level security kept its tenant-scoped keys inside their tenant, verified 8 for 8. A kill switch can halt all job dispatch fleet-wide. Gates built for hostile or fat-fingered humans turn out to be exactly what you want around an enthusiastic robot.
Derived state over memory. Anything the agent “remembers” is treated as suspect. What needs deploying is derived from git markers, not from the AI’s recollection of what it did. It once double-flagged an already-shipped change from memory; deriving the answer ended that class of error.
Cleanup is part of the run. The report’s last section is “state I left behind”: test keys revoked, test groups and baselines deleted, settings reverted, one honest line about what’s still uncommitted. Trust the ledger, not the intention.
What was missing (found the same way)
I asked the same agent to audit its own guardrails while drafting this post. It found two gaps, which is about the most on-brand ending this story could have.
First, its own cached auth tokens were sitting in the repo root, untracked but not gitignored. Night one’s fix had enumerated the two token files that existed; night two minted two new ones under different names and the list was already stale. That’s now a pattern rule instead of a list. Notice it’s the same bug class as the empty Domain Controllers group above: an enumeration that silently goes stale is how things fail politely.
Second: the harness scripts take a base URL, and nothing structural stopped one from being pointed at production. That was protected by convention, and the whole lesson of this system is that discipline goes in gates, not in promises. So there’s now a guard module every runner imports: point the harness at a production hostname and it exits with a refusal. The override isn’t a flag, it’s a full sentence you have to set in the environment, because consent should cost more than a typo. That applies to the robot’s tooling exactly as much as to the product.
If you’re thinking about doing this
The agent didn’t replace testing discipline. It made testing discipline affordable at 2am, at a depth I would never sustain by hand: 48 group-rule fields swept, 20 job actions against real Windows, every hub, every severity path, the billing flow with real test cards.
But the order of operations is the entire trick. Blast radius, identity, gates, then autonomy. Decide what the worst night looks like before the first night, give the thing its own name in your audit log, and keep the promote button human.
Then let it run while you sleep. The bug it finds at 1am was already in your product at 1am. The only question is who meets it first.
The short version
The AI didn’t replace my QA process. It gave me an overnight engineer that executes the boring depth, investigates its own failures, and keeps going long after I’ve gone to bed.
But it was wrong often enough that I wouldn’t trust it without isolation, its own identity in the audit log, approval gates, reproducible evidence, and a hard production boundary.
Autonomy is useful. Unbounded autonomy is just an unattended incident.