-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Server Security
-
Fully Compatible
-
ALL
-
Server Security 2026-10-23
-
None
-
None
-
None
-
None
-
None
-
None
-
None
If shutdown cancels the KMIP retries during the startup checkpoint install, a disaggregated node can fassert (10985542) instead of shutting down cleanly.
What happens
- loadKey declines with ECANCELED (13449402), so the checkpoint install is abandoned and KEKs are never loaded into the cache.
- CheckpointManager::setStartupCheckpoint() sees the shutdown status from _installCheckpoint() (13035200), skips its fassert, and discards the status because it returns void.
- LogServerRunnerStandbyImpl::setStartupCheckpoint() logs "Successfully set startup checkpoint" (10985523) and returns OK, so startup starts the standby loop (11757201).
- LogServerManager's shutdown token is not cancelled until the SLS state machine shuts down, which comes after the KMIP cancellation. If startup wins that race, the standby loop reads the log from the segment start and reaches a checkpoint-end marker. Verifying it needs a KEK that is not in the cache (11358100), so CheckpointManager::receivedCheckpointEnd() fasserts with 10985542 ("Checkpoint completion marker did not pass integrity check").
Seen in kmip_shutdown_during_checkpoint_pickup.js on Evergreen. The test's own assertions passed; the task failed because a core dump was found. Whether a run crashes is timing-dependent: in another run the shutdown cancelled the token first and the loop exited cleanly. The race was introduced by the shutdown decline added in SERVER-135226. It does not depend on how quickly the KMIP retry delay wakes on cancellation.
Proposed fix
- Make CheckpointManager::setStartupCheckpoint() return Status. Return the shutdown error instead of discarding it, and keep the fassert (13035201) for any other error.
- Propagate that status from LogServerRunnerStandbyImpl::setStartupCheckpoint(). initialise() then fails, and Initial::accept() stays in kInitial without starting the standby loop. Shutdown cancels the startup AsyncTry token before it cancels the KMIP retries, so this failure is not retried.
- Use uassertStatusOK at the offline-validation caller in disaggregated_service_lifecycle.cpp.
- Give CheckpointManagerMock a default Status::OK() action for setStartupCheckpoint.
- Unit tests: setStartupCheckpoint() returns the shutdown error when the pickup is abandoned or when the post-load callback is interrupted, and the standby runner returns that error.
- In kmip_shutdown_during_checkpoint_pickup.js, fail if the node logs "Fatal assertion" or "Got signal", so a crash fails inside the test instead of only at the core-dump check.
A prototype of this fix passed the CheckpointManagerTest, CheckpointManagerTestDeathTest and log_server_runner_standby_impl_test unit tests.
- is related to
-
SERVER-135226 Node aborts on invariant when KMIP decryption is cancelled by shutdown during checkpoint pickup
-
- Closed
-