Overview

Killing the primary during startup (or during a seal) can leave the disaggregated-storage cluster
with no primary at all, and the killed node never rejoins as SECONDARY. Observed in 3 of 20
runs of disagg_pali_chaos_key_long on unmodified master at 59f9dfe6691
(atlas-amazon2023-arm64).

Filed to Server Security for triage/repro and onward assignment.

This is distinct from SERVER-135302 (chronic standby lag). The lag watchdog did not fire in any of
these three runs; the cluster loses its primary outright rather than losing a lagging standby.

Symptom

Run 28 – no candidate steps up

\[CHAOS\] \[FRAMEWORK\] Failover did not complete for kill\_primary\_mid\_startup2; stopping chaos immediately
\[FRAMEWORK\] Injection failed for kill\_primary\_mid\_startup2 \(evt\-080\):
  \[FAILOVER\_DID\_NOT\_COMPLETE\] assert.soon failed \(timeout 120000ms\), msg : No candidate stepped up to PRIMARY

\[MONGOD BUG\] worker 1 wedged: 165418ms of consecutive failures from T\+2314846ms \(errors={"133":3,"515":1}\)
\[MONGOD BUG\] worker 4 wedged: 165545ms of consecutive failures from T\+2314719ms
\[MONGOD BUG\] worker 5 wedged: 165459ms of consecutive failures from T\+2314805ms
\[MONGOD BUG\] worker 6 wedged: 165514ms of consecutive failures from T\+2314750ms
\[MONGOD BUG\] worker 2/3/7 wedged: \~135790ms of consecutive failures from T\+2315405ms \(errors={"133":4}\)

\[MONGOD BUG\] Primary did not become writable within 30s after chaos phase
\[MONGOD BUG\] validate: Primary unreachable: Timed out waiting for any of \[20092,20093\] to become primary after 60000ms
\[MONGOD BUG\] ack'd writes: primary: unreachable: ... after 60000ms
\[MONGOD BUG\] lost update check: primary unreachable for update verification: ... after 60000ms
\[MONGOD BUG\] dbHash: primary unreachable for dbHash: ... after 60000ms
\[MONGOD BUG\] No primary found for post\-chaos encryption validation

All 7 workers wedged for 135-165 seconds. Error codes: 133 FailedToSatisfyReadPreference,
515 ConnectionClosedByPeer.

Run 29 – killed node crash-loops instead of rejoining

\[CHAOS\] \[FRAMEWORK\] Deferred restart failed after kill\_primary\_mid\_startup2:
  assert.soon failed \(timeout 90000ms\), msg : Restarted node at port=20093 did not reach SECONDARY within 90000ms
\[MONGOD BUG\] node did not rejoin as secondary within 90s after kill\_primary\_mid\_startup2 \(evt\-039\);
  it is crash\-looping or hung \-\- see its log
\[MONGOD BUG\] Primary did not become writable within 30s after chaos phase
\[MONGOD BUG\] validate: Primary unreachable: Timed out waiting for any of \[20092,20093\] ... after 60000ms

Run 32 – same, after a different event

\[CHAOS\] \[FRAMEWORK\] Deferred restart failed after kill\_primary\_then\_seal:
  assert.soon failed \(timeout 90000ms\), msg : Restarted node at port=20093 did not reach SECONDARY within 90000ms

Evergreen CLI command to retrieve test logs:

Patch 676 (run 28):

evergreen task build TestLogs --task_id 'mongodb_mongo_master_nightly_atlas_amazon2023_arm64_disagg_pali_chaos_key_long_patch_59f9dfe6691207c960400f56fc7bed3584fcae22_6aad8163430f680007247e1b_26_09_18_18_22_36' --execution '0' --log_path 'job0/45134587-8b71-45ec-8a6c-bce124370194.log' --logs_to_merge 'job0/global.log' --o output.txt

Patch 677 (run 29):

evergreen task build TestLogs --task_id 'mongodb_mongo_master_nightly_atlas_amazon2023_arm64_disagg_pali_chaos_key_long_patch_59f9dfe6691207c960400f56fc7bed3584fcae22_6aad816e74d4c000070a8f34_26_09_18_18_22_46' --execution '0' --log_path 'job0/629af4e5-6a28-49ea-b3c7-78bfded5e3fb.log' --logs_to_merge 'job0/global.log' --o output.txt 

Patch 680 (run 32):

evergreen task build TestLogs --task_id 'mongodb_mongo_master_nightly_atlas_amazon2023_arm64_disagg_pali_chaos_key_long_patch_59f9dfe6691207c960400f56fc7bed3584fcae22_6aad8191edc73b000700b190_26_09_18_18_23_22' --execution '0' --log_path 'job0/49b20fdd-5454-46a7-ae3d-2a2f717c77b0.log' --logs_to_merge 'job0/global.log' --o output.txt

Why this matters

The cluster ends with neither node able to serve as primary: every post-chaos validation (validate, ack'd writes, lost-update check, dbHash, encryption validation) fails with "primary unreachable". A 165 s window with no writable primary is a hard availability failure, not a degraded-performance issue.

Suggested triage steps

Reproduce with the recorded seeds (see "Reproduction" below) and capture the killed node's full

log – the harness reports it as "crash-looping or hung" but does not say which.

Determine whether the surviving node refuses to step up, or whether no election is called at all

(No candidate stepped up to PRIMARY over a 120 s window suggests the latter).

Check whether kill_primary_mid_startup2 leaves durable state that blocks the restarted node

from reaching SECONDARY – both events that trigger this kill the primary at a specific point
(mid_startup, then_seal), so a partially-initialised on-disk state is a plausible lead.

Reproduction

disagg_pali_chaos_key_long on atlas-amazon2023-arm64 at commit 59f9dfe6691.
Observed in 3 of 20 runs. Shell seeds: run 28 and run 32 seeds are recorded in their task logs;
run 29's chaos event was evt-039, run 28's evt-080.

Acceptance criteria

• Root cause identified for (a) no candidate stepping up and (b) the restarted node failing to
  reach SECONDARY; these may or may not be the same defect.
• A killed primary either triggers a successful election within the failover budget, or the failure
  to do so is explained and bounded.
• The restarted node rejoins as SECONDARY within the 90 s budget, or the reason it cannot is
  understood and surfaced in its log rather than reported as "crash-looping or hung".
• No post-chaos validation step reports "primary unreachable" in a run whose injected faults are
  individually survivable.

Notes

• Two further failure families from the same 20 runs are tracked separately: SERVER-135302 (chronic
  standby lag, 10 of 15 failures) and the unexplained-write-outage ticket filed alongside this one.
• Run 29 exhibits both this failure and the unexplained write outages, so the two may share a cause.
• The loadKey: cannot load encryption keys fatal (id 13479400) seen in these runs is expected
  and does not fail rounds – SERVER-134794 working as designed.