A deploy silently broke everyone’s MFA - and what it taught us about encrypting the Data Protection key ring

ASP.NET Core’s Data Protection stack is one of those things that “just works” until the day it spectacularly doesn’t. Here’s the three-layer hole we found in ours, and the live migration it took to close it properly.

Layer 1: the key ring evaporated on every deploy

We host on an ephemeral container platform (Render). Out of the box, Data Protection writes its key ring to the local filesystem - which gets wiped on every deploy and container rebuild.

The symptom was eerie: a user with the correct password and the correct TOTP code would fail to log in. Nothing was wrong with their credentials. The problem was that we use IDataProtector to encrypt MFA secrets and stored credential passwords, and after a deploy the key that had encrypted them was gone. Same data, same code, but the cipher key had been regenerated out from under us, so every decrypt failed.

Fix: persist the key ring somewhere durable - for us, Postgres.

builder.Services.AddDataProtection()
    .SetApplicationName("RFF.Api")
    .PersistKeysToDbContext<AppDbContext>();

Now the ring survives rebuilds. Crisis over - except we’d just created the next one.

Layer 2: the key ring was sitting in the database in plaintext

PersistKeysToDbContext stores the keys as plaintext XML. So the master keys that protect every credential password and MFA secret in the system were now sitting in a table, in the clear. Anyone who could read the database - a backup leak, a misconfigured replica, an over-broad support query - could lift those keys and decrypt everything.

The fix is to wrap the ring with an X.509 certificate so the keys are stored encrypted:

var cert = new X509Certificate2(
    Convert.FromBase64String(pfxBase64), pfxPassword,
    X509KeyStorageFlags.EphemeralKeySet); // private key in memory only - read-only container FS

dpBuilder.ProtectKeysWithCertificate(cert);

Two details worth stealing:

EphemeralKeySet keeps the private key in memory instead of trying to write it to a user key store that doesn’t exist on a read-only container filesystem.
Fail closed in production. If the wrapping cert isn’t configured, we refuse to boot in prod rather than silently fall back to a plaintext ring and quietly re-open the hole:

else if (builder.Environment.IsProduction())
    throw new InvalidOperationException(
        "Refusing to start in Production with an unencrypted Data Protection key ring.");

Flip the cert on, redeploy, done - right? Not even close.

Layer 3: turning on encryption protects nothing you already have

This is the part that surprised me, and it’s the reason a lot of “we added ProtectKeysWithCertificate” setups are quietly still exposed.

ProtectKeysWithCertificate only encrypts keys created after you switch it on. Your existing key - the plaintext one - stays in the ring and stays the active encryptor until it expires (~90 days). So immediately after the change:

Every value already in the DB is still under the plaintext key. Unchanged.
Worse, newly protected data also lands under the old plaintext key. Data Protection won’t promote a brand-new key to default encryptor until it clears a ~2-day propagation window (designed for multi-server rollouts), so during that window the immature new key is ignored and the plaintext key keeps winning.

Flipping the config flag felt like a fix but changed nothing on disk. To actually cut over, we had to force a key rotation and re-encrypt the existing data - carefully, on a live database, without a window where a value is readable by nobody.

The cutover: rotate, re-encrypt, verify before you destroy anything

We wrote a one-shot startup routine. It runs before the app serves traffic, gated behind a config flag and an idempotency marker so it can never run twice. The ordering matters:

// 1. New key, cert-wrapped (because the cert is now configured), active *now*
//    (back-date activation so it isn't stuck "immature").
keyManager.CreateNewKey(activationDate: now.AddMinutes(-5),
                        expirationDate: now.AddDays(90));

// 2. Revoke every old key. THIS is what defeats the ~2-day propagation window -
//    revoked keys leave the default-encryptor candidate set, so the new key wins now.
foreach (var keyId in oldKeyIds)
    keyManager.RevokeKey(keyId, "Rotated to certificate-wrapped key.");

// 3. Read each value with DangerousUnprotect (ignoreRevocationErrors: true - we just
//    revoked the key that encrypted it) and re-Protect under the new key.
var plain      = DangerousUnprotectToString(protector, value);
var reprotected = protector.Protect(plain);

// 4. VERIFY each value round-trips through the NORMAL, revocation-honoring Unprotect,
//    entirely in memory, BEFORE SaveChanges. If anything fails to land under a live
//    key, we throw and persist nothing.
if (protector.Unprotect(reprotected) != plain)
    throw new InvalidOperationException("Re-encryption verification failed.");

// 5. Only after the verified SaveChanges: delete the old plaintext key rows.
// 6. Stamp an idempotency marker so this never runs again.

The two things that make this safe:

Revoke-then-DangerousUnprotect. Revoking the old keys is what forces the new key to become the default encryptor immediately. But revoked keys can’t be read by the normal Unprotect - so to read the existing data one last time we use DangerousUnprotect(ignoreRevocationErrors: true). It’s “dangerous” precisely because it ignores revocation; using it deliberately, for exactly this migration read, is the correct use.
Verify-before-destroy. Every value is re-encrypted and round-tripped through the strict Unprotect in memory first. Only once all of them provably decrypt under a live key do we SaveChanges, and only after that do we delete the plaintext keys. There is no moment where data is committed in an unreadable state.

One more subtlety we had to respect: the key-store writes in steps 1-2 don’t touch the protector’s in-memory key-ring cache - the cache is built lazily on the first protect/unprotect call. So all key mutation has to happen before the re-encrypt loop, while nothing has materialized the ring yet. Running at startup, pre-traffic, guarantees that.

Takeaways

On any ephemeral/PaaS host, persist your Data Protection key ring or you’ll silently invalidate everything it protects on the next deploy. (If you encrypt MFA secrets with it, that’s your MFA.)
Persisting ≠ protecting. A persisted ring is plaintext by default. Wrap it (ProtectKeysWithCertificate) and fail closed if the cert is missing in prod.
Turning on encryption doesn’t migrate existing data, and the propagation window means new data isn’t even safe right away. A real cutover means rotating the key, re-encrypting, and verifying before you delete the old keys.

That last rule - verify before you destroy - shows up on the other end of the wire too. The agent seals its own credential at rest with Windows DPAPI and round-trips the decrypt before it blanks the cleartext; the one-line enrollment story covers that side.

We’re building RFF - hosted fleet management for Windows. We store admin credentials and MFA secrets, so this stuff isn’t optional; here’s how we secure it. Free for your first 100 endpoints.