A disaggregated-storage mongod crashes unconditionally when the WiredTiger Checkpointer thread runs a checkpoint while the page materializer is unavailable, and WiredTiger encounters WT_NOTFOUND when trying to remove an entry from the shared stable timestamp table (WiredTigerShared.wt_stable).

In WiredTiger's layered (disaggregated) storage, `WiredTigerShared.wt_stable` is a shared table that records stable timestamps across the cluster. During a checkpoint, WiredTiger calls `_wt_btcur_remove` to remove or update an entry in this table. When the page materializer has been killed, the shared table may be in an inconsistent or partially-updated state, and the expected entry is absent, causing `_wt_btcur_remove` to return WT_NOTFOUND (-31803).

WiredTiger currently treats this WT_NOTFOUND as a fatal checkpoint failure and calls `__wt_abort`, crashing the entire WiredTiger connection and the mongod process with it. Both the new primary (d20061) and the restarted secondary (d20060) crash independently within ~2 minutes of each other on the same error, making the cluster irrecoverable until manual intervention.

The checkpoint should either tolerate WT_NOTFOUND from this remove (if the remove is idempotent by design) or fail the current checkpoint attempt with a recoverable error rather than aborting the entire WiredTiger connection.

Sequence of events

1. Chaos test injects: evt-019 kill_standby_page_materializer (unplanned) target=pagematd at T+347369ms. The page materializer process is killed.
2. Following the kill, the log server on one of the cellC zones (172.17.0.1:20047) becomes  unreachable ("Connection refused"), as seen in d20061's reconnect attempts at 20:41:38.637.
3. d20061 (now primary after the primary swap earlier in the test) is running its periodic WiredTiger checkpoint on the Checkpointer thread.
4. The checkpoint calls session.checkpoint("use_timestamp=true") which descends into __wt_checkpoint_db. During checkpoint, WiredTiger calls __wt_btcur_remove on a cursor over file:WiredTigerShared.wt_stable to remove an entry. Because the page materializer is down and the shared table state is stale, the entry is not found: WT_NOTFOUND (-31803).
5. WiredTiger logs WT_NOTFOUND from multiple call sites, each repeated multiple times as WiredTiger retries.:
   • __wt_btcur_remove (bt_cursor.c:1321)
   • __curfile_remove  (cur_file.c:566)
6. WiredTiger calls __wt_abort ("aborting WiredTiger library"), which crashes the process.
7. d20060 (restarted as secondary after its earlier crash) hits the same abort at 20:43:35 UTC under the same conditions.

Crash evidence

Log file: palichaoskek3.log.txt (attached)
Run date: 2026-06-26
Seed: 1782506123

Chaos event (line 1099725):

[CHAOS] Injecting: evt-019 kill_standby_page_materializer (unplanned) target=pagematd at T+347369ms compound=true 

d20061 crash (line 1068614, 20:41:38.596 UTC): WiredTiger errors immediately before SIGABRT (Checkpointer thread):

{"s":"E","c":"WT","id":22435,"ctx":"Checkpointer", "message":{"session_dhandle_name":"file:WiredTigerShared.wt_stable","session_name":"WT_SESSION.checkpoint","msg":"int __wt_btcur_remove(WT_CURSOR_BTREE *, ...):1321:Error at src/btree/bt_cursor.c:1321: \"WT_NOTFOUND\" failed", "error_str":"WT_NOTFOUND: item not found","error_code":-31803}} 

Stack trace (d20061, lines 1155060–1155122):

  #17  mongo::WiredTigerKVEngine::_checkpoint(WiredTigerSession&, bool)
       at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:2196
       [calls session.checkpoint("use_timestamp=true")]
  #18  mongo::WiredTigerKVEngine::_checkpoint(WiredTigerSession&)
       at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:2294
  #19  mongo::WiredTigerKVEngine::checkpoint()
       at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:2309
  #20  mongo::StorageEngineImpl::checkpoint()
       at src/mongo/db/storage/storage_engine_impl.cpp:962
  #21  mongo::Checkpointer::run()
       at src/mongo/db/storage/checkpointer.cpp:116 

Log server connection refused after page materializer kill (d20061, 20:41:38.637 UTC):  

{"s":"E","c":"DISAGG","id":11425608,"ctx":"Disagg-6",    "msg":"Failed to establish connection to log server",    "attr":{"logServer":"172.17.0.1:20047","grpcErrorCode":14,            "grpcErrorMessage":"... Connection refused"}} 

d20060 second crash (line 1309771, 20:43:35.573 UTC):   Same WiredTiger abort; triggered by the same underlying condition on the secondary. 

{"s":"E","c":"WT","id":22435,"ctx":"Checkpointer",    "message":{      "session_dhandle_name":"file:WiredTigerShared.wt_stable",      "session_name":"WT_SESSION.checkpoint",      "msg":"void __wt_abort(WT_SESSION_IMPL *):29:aborting WiredTiger library"}} 

Expected behavior

`__wt_btcur_remove` on `WiredTigerShared.wt_stable` returning WT_NOTFOUND during a checkpoint should not be treated as a fatal condition. Two acceptable resolutions:

• If the remove is idempotent (the goal is to ensure the entry does not exist after the checkpoint), then WT_NOTFOUND should be treated as a no-op and the checkpoint should continue. The desired post-state is already achieved.
• If the missing entry represents a genuine inconsistency that WiredTiger cannot resolve locally, the checkpoint should fail with a recoverable error (e.g., WT_ROLLBACK or a layer-specific error) that lets the MongoDB layer log the failure and retry, rather than calling __wt_abort and killing the entire process.

In neither case should a WT_NOTFOUND from a table remove during checkpoint abort the WiredTiger connection unconditionally. The page materializer can be unavailable transiently (e.g., killed and restarted by chaos or by a rolling restart), and the cluster must remain operable during such events.

Why this occurs in pali_chaos_kek.js but not pali_chaos.js

pali_chaos.js uses a static KEK via `encryptionKeyFilePath` and performs no KEK or CMK rotations. `featureFlagKEKPushMode` is false (default), so WiredTiger operates in pull mode: keys are fetched on demand via a `get_key` callback and are never written to `WiredTigerShared.wt_stable` at reserved timestamps.

pali_chaos_kek.js uses dynamic KEK generation backed by a PyKMIP server with
`featureFlagKEKPushMode: true`. In push mode, MongoDB commits each new KEK to the WiredTiger shared stable table (`WiredTigerShared.wt_stable`) at a reserved oplog timestamp via `pushToWT()`. As the rotation driver thread fires KEK rotations, successive keystore entries accumulate in this table.

During checkpoint, WiredTiger's layered storage code removes old superseded keystore entries from `WiredTigerShared.wt_stable` via `__wt_btcur_remove`. This cleanup path only runs in push mode because only push mode writes keystore entries to the table. When `kill_standby_page_materializer` is injected, the page materializer that keeps this table's pages materialized is killed, leaving the table's pages in an incompletely materialized state. The next checkpoint's remove call then returns WT_NOTFOUND for an entry it expected to find.

Since pali_chaos.js never writes keystore entries to `WiredTigerShared.wt_stable` (pull mode has no such writes), the `__wt_btcur_remove` cleanup path for these entries is never reached, and `kill_standby_page_materializer` cannot trigger this crash in pali_chaos.js.

Affected versions
-----------------
Observed on: current master (branch chyelin/SERVER-129742-chaos, 2026-06-26)
Feature flags: featureFlagKEKPushMode enabled (test parameter), featureFlagCMKRotation enabled

See also

Related test (SERVER-129742):

src/mongo/db/modules/atlas/jstests/pali_chaos/pali_chaos_kek.js

Crash path:    

• src/third_party/wiredtiger/src/btree/bt_cursor.c:1321 (__wt_btcur_remove)
• src/third_party/wiredtiger/src/cursor/cur_file.c:566  (__curfile_remove)
• WiredTigerShared.wt_stable (shared stable timestamp table)

Stack entry:  

src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:2196 (WiredTigerKVEngine::_checkpoint, session.checkpoint("use_timestamp=true"))