-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker, Test Format
-
None
-
Storage Engines - Foundations
-
None
-
None
Description:
In the multi-node predictable replay test, both leader and follower independently execute the same deterministic sequence of operations rather than the follower replaying only operations that succeeded on the leader. This exposes an asymmetry in clayered_remove_follower __ vs __ _clayered_remove_leader that causes spurious hash mismatches.
The scenario:
A REMOVE operation at replay_ts T targets a key whose update chain contains an UPDATE at commit_ts T_write, followed by an older tombstone. The follower's read_ts happens to fall below T_write but above the tombstone.
- Leader path ({}clayered_remove_leader): calls cursor->remove() directly on the stable table, which goes through }}{{wt_btcur_remove → __curfile_update_check detects the invisible committed UPDATE in the per-slot update chain → returns WT_ROLLBACK. test/format retries with a higher read_ts, eventually sees the UPDATE, and the remove succeeds.
- Follower path ({}clayered_remove_follower): calls }}{_}{{_clayered_lookup first, which checks visibility at the current read_ts. The UPDATE at T_write is invisible; the older tombstone is visible → the lookup returns WT_NOTFOUND. The follower treats this as "key already absent" and commits a no-op without retrying.
Result: The leader commits a tombstone at replay_ts T; the follower does not. The databases diverge and the hash comparison asserts.
Root cause: _clayered_remove_follower uses a visibility-based lookup __ (_clayered_lookup) instead of write-conflict detection. When an invisible committed update sits above a visible tombstone in the chain, it incorrectly returns WT_NOTFOUND rather than WT_ROLLBACK. See the related issue WT-17247, which describes a similar false-positive visibility case in the follower write path. This is the corresponding false-negative case.
Example patch build: https://spruce.corp.mongodb.com/version/69e96cefba54350007a03e2f/tasks