bug(backups): MongoDB to-copy restore leaves the target cluster unauthenticatable while RestoreJob reports Succeeded

[lexfrei]

What happens

A RestoreJob that restores a MongoDB backup into a second MongoDB application (the to-copy flow documented in examples/backups/mongodb) reports status.phase: Succeeded, but the target cluster is unusable afterwards. Every system user in the target now carries the source cluster's password, so the target's own <release>-percona-server-mongodb-users and internal-<release>-users Secrets no longer authenticate. The psmdb operator itself is affected: its clusterAdmin and clusterMonitor connections start failing at the moment the restore completes and never recover.

Evidence

The e2e case mongodb-2-backup-roundtrip has failed in 20 out of 20 CI runs since it landed, always at the same point: the final step that reads the restored sentinel document back out of the target.

== Step 30 verify: the sentinel round-tripped through S3 into the copy ==

MongoServerError: Authentication failed.
command terminated with exit code 1

The target cluster's mongod log gives a sharp boundary. The Cozystack RestoreJob was created at 15:47:37, the psmdb operator's pbm restore ran from 15:47:44 to 15:47:53, and the PerconaServerMongoDBRestore reached state: ready.

15:47:01 .. 15:47:50   every auth succeeds (clusterAdmin, clusterMonitor, userAdmin, backup)
15:47:44 .. 15:47:53   pbm logical restore replays the source dump into the target
15:47:52 onwards       every auth fails, and keeps failing to the end of the log

Each failure carries the same reason:

"msg":"Failed to authenticate","attr":{"user":"clusterAdmin","db":"admin",
"error":"AuthenticationFailed: SCRAM authentication failed, storedKey mismatch"}

storedKey mismatch means the user documents are present but their stored SCRAM credentials no longer match the ones the target's Secret holds. The failing set includes databaseAdmin from a mongosh 2.5.10 client, which is the example harness reading the sentinel back, and clusterAdmin and clusterMonitor from the operator itself.

Both halves before this point are genuinely healthy. The BackupJob completed in 75 seconds with BackupCompleted, the RestoreJob completed in 20 seconds with RestoreCompleted, and the sentinel data did travel through S3 into the target. The only thing that breaks is the ability to talk to the target afterwards.

Mechanism

A psmdb logical restore replays the source dump including the admin database, so the source cluster's admin.system.users documents replace the target's. Percona documents this precondition for restoring into a new environment: the destination must already hold a Secrets object with the same user passwords as the original cluster.

The Cozystack MongoDB chart cannot satisfy that precondition. packages/apps/mongodb/templates/mongodb.yaml hardcodes the users Secret to the release name:

  secrets:
    users: {{ .Release.Name }}-percona-server-mongodb-users

There is no value that overrides it, and the operator generates a fresh random password set per release. Two MongoDB applications in the same namespace therefore always disagree on system-user credentials, and a to-copy restore between them always ends with the target holding credentials nobody can look up through the target application.

Impact

• A tenant following examples/backups/mongodb end to end is left with a broken target cluster and a Succeeded RestoreJob telling them it worked.
• The target's <release>-credentials Secret, which is what the dashboard surfaces as the connection URI, is silently stale.
• The psmdb operator loses its own access to the target, so subsequent reconciliation of that cluster is degraded.
• mongodb-2-backup-roundtrip holds the in-tree e2e suite red on every branch that runs it.

Notes on the fix space

I am not proposing a specific fix, but the shape of the problem constrains it. The restore path in internal/backupcontroller/psmdbstrategy_controller.go builds the operator Restore CR from backupSource plus clusterName and does not touch system users, so any fix has to decide deliberately what the target's credentials should be after a cross-cluster restore. Making the target adopt the source's system-user Secret before the restore matches what the operator expects, but it hands one application's credentials to another and that is a security decision rather than a mechanical one. Reporting the RestoreJob as Succeeded while the target is unreachable is worth treating as a separate defect from the credential handling itself.

One thing this is not: the bootstrap gate for the default MongoDB strategy is not involved. cozy-default-mongodb was present and the BackupJob dispatched against it normally in every failing run.

Reproduction

Run examples/backups/mongodb/run-all.sh in a namespace with a provisioned bucket, or run the mongodb chainsaw suite from hack/e2e-chainsaw/mongodb/. It fails on the final verification every time.

[dosubot]

The diagnosis is accurate. Here's what the code confirms and where the fix surface is:

Root cause confirmed in the chart: mongodb.yaml hardcodes spec.secrets.users: {{ .Release.Name }}-percona-server-mongodb-users with no override [1], and user-secrets.yaml generates fresh random passwords via randAlphaNum on first install [2]. Two MongoDB releases therefore always disagree on system-user credentials.

The mongosh_exec helper reads from internal-<cr>-users [3] — after a logical restore replays the source's admin.system.users documents, the target's Secret still holds its own passwords, producing the storedKey mismatch.

Two separable fixes:

1. The credential mismatch — the restore controller already re-points S3 credentials at the target's own secret, but makes no analogous move for the users secret. A pre-restore step in reconcileMongoDBRestore could copy the source's <source>-percona-server-mongodb-users Secret over the target's before creating the PerconaServerMongoDBRestore CR. Alternatively, after the restore completes, generate fresh passwords into the target's users secret and let the operator reconcile them via db.changeUserPassword() — this is the same mechanism the upgrade path already uses. The latter avoids persisting the source's credentials into the target release, at the cost of a second operator reconcile cycle before the cluster is usable.

2. The false Succeeded status — the controller marks RestoreJobPhaseSucceeded the moment PerconaServerMongoDBRestore reaches state: ready [4], with no post-restore health check. A liveness probe (e.g. attempt a databaseAdmin auth against the target before reporting success) would catch the broken cluster and flip the phase to Failed with an actionable message.

Harness note: mongosh_exec reads credentials immediately after RestoreJob reaches Succeeded [3], so any fix that relies on a post-restore operator password rotation also needs a readiness gate in the harness before the sentinel verification step.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[lexfrei]

Reproduction update: with #3834 the mongodb chainsaw suite stops after the ad-hoc BackupJob succeeds, so it no longer reaches the restore leg and stops reproducing this. To reproduce, run examples/backups/mongodb/run-all.sh without SKIP_RESTORE; the full to-copy flow is still there and still fails on the final sentinel read.

One more place that describes the restore leg as covered: docs/operations/backup-classes.md says the mongodb example "asserts the round-trip while the source stays untouched". That sentence predates #3834 and is not touched by it. Whoever fixes this issue should sweep that line together with re-enabling the suite's restore leg; until then the example README is the authoritative statement of where coverage ends.