Overview

When every CMK is unavailable, a standby that must unwrap its KEK during disaggregated checkpoint
pickup aborts on an invariant (SIGABRT + core) instead of failing startup cleanly with a non-zero
exit code.

Bug / symptom

disagg_pali_chaos_key_long (jstest pali_chaos_key.js, event {{evt-028
restart_standby_under_kms_outage}}) failure. The standby is deliberately restarted while PyKMIP
reports every CMK as revoked. Instead of exiting, mongod aborts:

"s":"F","c":"ASSERT","id":23083,"ctx":"Disagg\-0","msg":"Invariant failure",
"attr":{"expr":"ret","error":"BadValue: 22: Invalid argument \- ",
"location":"src/mongo/db/storage/wiredtiger/wiredtiger\_kv\_engine.cpp:2345:31:
 virtual Status mongo::WiredTigerKVEngine::setRecoveryCheckpointMetadata\(std::string\_view\)"}
"s":"F","c":"CONTROL","id":6384300,"msg":"Writing fatal message",
"attr":{"message":"Got signal: 6 \(Aborted\).\n"}

The chaos event documents the contract it violates – _"A KEK it cannot unwrap must exit startup,
not abort: the expected outcome is a non-zero exit having logged nothing fatal"_ – and its
readFatalLines guard exists precisely so that a product assertion is not absorbed by the
event's intentional crash. That guard fires recordUnexpected and fails the round.

Root cause

The Status describing the real, expected failure is flattened to a bare EINVAL at the
WiredTiger boundary, where it becomes indistinguishable from a programming error, so the storage
layer treats it as fatal.

setKMIPOutage revokes CMK UIDs 2 and 5; the standby restarts with no cached KEK.

During startup checkpoint pickup, loadKey calls

encryptorManager->initializeFromExistingCluster(), which fails with
BadValue: KMIP decrypt failed, code: 10 error: UID 5 has been revoked (failure injection).
Both CMKs fail; retry is correctly skipped (willRetry: false) since revocation is not
transient from the client's point of view.

key_provider_callbacks.cpp:94 logs that Status and returns a bare EINVAL

The status is discarded. KeyProviderHandle (key_provider_callbacks.cpp:22) has nowhere to record
it.

WT propagates 22 up through __wti_disagg_load_crypt_key ->

*disagg_pick_up_checkpoint ("Failed to load encryption keys for checkpoint
7686548934944097283") -> *wti_disagg_conn_config -> WT_CONNECTION::reconfigure.

WiredTigerKVEngine::setRecoveryCheckpointMetadata

(wiredtiger_kv_engine.cpp:2331-2346) special-cases only EBUSY (returned as
ObjectIsBusy, added by SERVER-132066) and falls through to invariantWTOK(ret, nullptr)
for everything else. EINVAL hits the invariant and the process aborts.

The caller (CheckpointManager::_updateStorageCheckpoint, checkpoint_manager.cpp:643)
therefore never gets the chance to uassertStatusOK its way to the clean startup failure the
design calls for.

Proposed fix

Preserve the key-provider failure Status across the WT boundary and make the checkpoint-pickup
path non-fatal for it, without weakening the invariant for genuine EINVAL programming
errors.

Add a synchronized Status lastKeyProviderError to KeyProviderHandle. Every

return EINVAL site in loadKey (and the sibling callbacks) records the Status it
currently only logs, and clears it on a successful load.

In WiredTigerKVEngine::setRecoveryCheckpointMetadata, when reconfigure returns

EINVAL, consult the recorded key-provider Status. If one is set, return it (with context
that the checkpoint could not be picked up) instead of invarianting. If none is set, keep the
existing invariantWTOK – an EINVAL with no key failure behind it is still a bug.

Confirm the resulting uassertStatusOK at checkpoint_manager.cpp:643 produces a clean

non-zero startup exit, and that the steady-state caller at checkpoint_manager.cpp:719
either retries or shuts the node down cleanly rather than aborting.

Alternative, if preserving the Status proves impractical: have WT return a distinct error code
for crypto-key load failure so the engine can classify it directly. The Status-passing approach
is preferred because it keeps the KMIP diagnostics in the error the operator actually sees.

Acceptance criteria

• A standby started with every CMK revoked exits startup with a non-zero exit code, logs no
  Invariant failure / Fatal assertion / Got signal, and produces no core dump.
• The surfaced error names the underlying KMIP cause (revoked/unavailable CMK), not a bare
  "Invalid argument".
• setRecoveryCheckpointMetadata still invariants on an EINVAL that is not attributable to
  a key-provider failure.
• New unit test in wiredtiger_kv_engine_test.cpp: key-provider failure recorded + EINVAL
  => Status returned; no recorded failure + EINVAL => invariant.
• kmip_outage_startup.js extended (or a new deterministic jstest added) to cover the startup
  checkpoint-pickup path with all CMKs revoked, asserting a clean exit.
• restart_standby_under_kms_outage completes without recordUnexpected, and
  disagg_pali_chaos_key_long passes a full multi-round run.

Technical notes

• Only the startup (isDiscontinuous) caller at checkpoint_manager.cpp:643 is confirmed by
  the failing logs. The steady-state caller at checkpoint_manager.cpp:719 has the same
  exposure but is unverified – a transient KMS outage there would abort a running node rather
  than retry, which would warrant a higher priority if confirmed.
• Related: SERVER-132066 (added the EBUSY retry branch this bug falls through),
  SERVER-131952 (KEK rotation vs. checkpoint ordering in the same callbacks).