-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Critical - P2
-
Affects Version/s: None
-
Component/s: DHandles
-
Storage Engines - Foundations
-
93.717
-
None
-
None
Problem
On a disaggregated-storage follower, a timestamped (or non-transactional) reader resolving the newest stable checkpoint is not atomic with the open of that checkpoint's dhandle: the OUTDATED check happens in the dhandle find (_wt_conn_dhandle_find), the session_inuse pin happens later in the open (wt_cursor_dhandle_incr_use), and nothing revalidates after the open. A thread suspended between the two for the duration of a checkpoint pickup opens a checkpoint that the pickup already retired: the merge marked its dhandle OUTDATED (wti_conn_dhandle_outdated) and the prune walk advanced the ingest prune_timestamp past it (_layered_update_ingest_table_prune_timestamp, which has no pinned-timestamp clamp). The reader then reads the old stable checkpoint while the ingest content covering the gap was pruned — a stale read: an old committed value is returned for a key updated between the two checkpoints.
Note this is not only an interleaving problem but a lack of synchronization: on the dhandle-reuse path there is no lock and no happens-before edge (no lock-free synchronization either) that would guarantee a reader observes OUTDATED, or that the prune walk observes the reader's pin — the protocol currently relies on ordering and timing rather than on any synchronizing primitive. The suspension interleaving is just how we make the missing synchronization observable; the gap exists regardless of it.
Only timestamped / non-transactional readers are exposed: untimestamped snapshots pin WT_GEN_DISAGG_CKPT and defer the adoption.
Reproduced
Deterministic reproducer in a comment below (temporary, not for commit): an env-armed sleep hook in __wt_session_get_dhandle suspends the reader's open after the find matched, before the inuse pin. The race run returns the stale value 5/5; the control run (reader starts after pickup+prune) returns the correct value.
Impact
Wrong (stale) read results on followers in an extreme scheduling interleaving. No committed data is lost.
Fix direction
Validate-after-open: acquire-load last_checkpoint_meta_lsn after binding the stable dhandle and retry the bind if it advanced past the resolved checkpoint (same pattern as __txn_snapshot_validate_disagg). Alternatively make the check-and-pin atomic via a dedicated lock around find→inuse. Independent of WT-18452's flag-atomicity fix; that fix alone does not close this hole.
Related
WT-18452 (unlocked OUTDATED update), WT-17772, WT-17968, WT-18408
- is related to
-
WT-17968 Disaggregated storage checkpoint pick-up pinned-timestamp panic check reads unpopulated metadata (dead code)
-
- Blocked
-
-
WT-18452 Unlocked WT_DHANDLE_OUTDATED flag update in __wti_conn_dhandle_outdated races dhandle close
-
- In Code Review
-
-
WT-18156 Follower checkpoint pickup breaks snapshot isolation for untimestamped readers
-
- Closed
-
-
WT-18408 Data loss when a picked up checkpoint's oldest timestamp > current read timestamp
-
- Closed
-
-
WT-18533 Read-only cursor scan can dirty a live stable btree across step-down - "disaggregated btree dirtied while not leader" assertion
-
- Needs Scheduling
-
-
WT-17772 Consider marking data handles outdated at step-down or step-up
-
- Open
-
-
WT-18542 Audit and document the WT_DHANDLE_OUTDATED lifecycle and its coupling with dhandle->session_inuse
-
- Needs Scheduling
-
- related to
-
WT-18452 Unlocked WT_DHANDLE_OUTDATED flag update in __wti_conn_dhandle_outdated races dhandle close
-
- In Code Review
-
-
WT-18533 Read-only cursor scan can dirty a live stable btree across step-down - "disaggregated btree dirtied while not leader" assertion
-
- Needs Scheduling
-
-
WT-18542 Audit and document the WT_DHANDLE_OUTDATED lifecycle and its coupling with dhandle->session_inuse
-
- Needs Scheduling
-
-
WT-18553 Investigate a durable design for the ingest prune timestamp calculation
-
- Needs Scheduling
-