The problem

When a node starts and cannot decrypt its checkpoint because the KMIP server is briefly unavailable, the node aborts. It does so again on every restart, for as long as the outage lasts, instead of becoming a secondary once KMIP answers. The outage need not be long or permanent — the same stall was ridden out successfully in one incident and aborted the node in another, and what separated them was whether the last permitted retry happened to succeed.

The abort also names no cause. The fatal message is an invariant failure reading BadValue: 22: Invalid argument, with no indication that an external dependency was the reason, so the log points at the storage engine rather than at the outage that triggered it.

Evidence

• AF-22233 (9.1.0-rc1023, 2026-09-23, 4 occurrences): KMIP decrypt retries exhaust against 127.0.0.1:35696 → 11800518 "loadKey: Failed to initialize from existing cluster" → the checkpoint pickup fails in WiredTiger → invariant failure in setRecoveryCheckpointMetadata → SIGABRT.
• AF-21485 (9.1.0-rc1021, Open, ~85 linked reports): the same sequence at the same function, line :2336 → :2345 between builds.
• HELP-100144 (9.1.0-rc1021, 2026-09-14): the same stall in which the retries did succeed — on the final attempt, ~84 s in — so the node survived and became a secondary. Same condition, opposite outcome. It is also a checkpoint pickup by a starting standby, not fresh-cluster initialization: the log lines it quotes (12376600, 11225600) come from the checkpoint pickup and standby startup functions, and its slow decrypt traces to the same key call site (key_provider->load_key).
• What separates the two paths is the KMIP operation, not the age of the cluster: the first step-up on a new cluster writes the KEK and goes through the fresh-cluster path with the long retry budget, while any starting node picking up an existing checkpoint decrypts the KEK through loadKey regardless of when the cluster was created. This report is the second case.
• Both AF tickets are node starts loading an existing checkpoint. This is not the fresh-cluster initialization window that SERVER-134823 and SERVER-134754 address: that path carries a 10-minute KMIP retry budget, while this path keeps the 75-second default.
• Not the case SERVER-132066 fixed. 132066 made this same invariant retryable for WiredTiger's transient EBUSY (data handles in use, superseded checkpoint) on the steady-state and step-up install paths, and deliberately left both the non-EBUSY errors and the startup install fatal — its commit records "other errors still invariant" and "Startup keeps its fassert: nothing can hold data handles before the catalog opens". That reasoning holds for a busy pickup, which really is a logic error at startup; it does not hold when the cause is an unavailable key server.

What happens

• At startup the node installs the last completed checkpoint from the log service, which requires decrypting the checkpoint's key material through KMIP.
• When that decryption fails after its retries, the storage engine reports the checkpoint pickup as EINVAL. The code is produced by the server's own key-provider callback and means "the key could not be loaded", not "bad argument".
• The failure is treated as impossible at two points: the checkpoint pickup treats EINVAL as a logic error and aborts, and the startup checkpoint install treats any non-shutdown failure as fatal. An unavailable key server reaches both.

History

• AF-21485 has carried a request to change this behaviour since 2026-09-11, confirmed feasible there on 2026-09-15. It was never ticketed: AF-21485 was declared covered by SERVER-134754 on 2026-09-22, and AF-22233 then crashed on the release containing that fix.

Related

AF-22233, AF-21485, HELP-100144, SERVER-132066 (retry on EBUSY in the same function), SERVER-134823, SERVER-134754, CLOUDP-441739

Existing coverage

jstests/disagg_storage/encryption/kmip_transient_outage_kek_retry.js, kmip_unreachable_before_replication.js