-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
Storage Engines - Foundations
-
9.207
-
None
-
None
No layer prevents a follower from adopting stable content whose oldest timestamp exceeds the read timestamp of an active transaction. The server explicitly permits readers behind the oldest timestamp and accepts a WiredTiger-side crash as the detection mechanism, on the grounds that the 30-minute history window makes the overlap rare (SERVER-118879). WiredTiger's side of that agreement, the WT-16467 pick-up panic, is currently dead code: the comparison in {}wt_disagg_pick_up_checkpoint _reads metadata.oldest_timestamp before }}{{_wt_disagg_parse_meta populates it (WT-17968), and the attempt to enable it is blocked on SERVER-122575 because the pinned-timestamp comparison false-positives on a stale local oldest timestamp during the rollout window.
When the overlap happens, the reader's data is unrecoverable by construction: the leader's reconciliation discards a key and its history once its tombstone is globally visible at the leader's oldest timestamp, so the new checkpoint contains neither the key nor a history store version serveable at the reader's timestamp. The consequences on the follower:
- An unpositioned timestamped reader silently loses the key: __clayered_can_advance_stable takes the read-timestamp fast path, reopens the stable cursor onto the new checkpoint, and a subsequent search returns WT_NOTFOUND for a key the same transaction read earlier. Reproducer: test_layered_pinned_read_adopted_oldest.py (fails on the current develop tip).
- A positioned reader is protected only by the guard
WT-17969hoisted above the fast path (see the FIXME-WT-17968 comment there), and only on the checkpoint pick-up path; the same condition reached through the role-change reopen (step-down) aborts on "upgrading a positioned stable cursor" (WT-18339).
Proposed solution
- Assert at stable-content adoption (checkpoint pick-up and role-change) that the incoming content's oldest timestamp does not exceed the read timestamp of any active transaction — the oldest is already in the adopted checkpoint metadata; the reader side is a scan of the active transactions' read timestamps. Deliberately not the pinned timestamp: pinned includes the connection's local oldest, which lags during the SERVER-122575 rollout gap and produced the false positives that got the WT-17968 panic reverted (along with the capped-pinned bug, draft PR #14149). A reader-only minimum tests exactly the unsafe condition — readers are a term of pinned, so it catches every case the pinned-based panic would — with no dependency on the server-side rollout, and it also covers the role-change adoption path the pick-up panic never did. This effectively supersedes the WT-17968 enablement path.
- With that invariant enforced, move the positioned-on-stable check in {}clayered_can_advance_stable _back inside the no-read-timestamp branch, as its FIXME anticipates. Under oldest <= read timestamp the move is safe: a key visible at the reader's timestamp cannot have its removal globally visible at the incoming oldest, so the key (or its history store version) survives and the position transfer in }}{{_clayered_reopen_stable succeeds; the reopen assertion becomes unreachable for timestamped readers.
Notes:
- The check races against a transaction publishing its read timestamp concurrently with the adoption-time scan (the same ordering family as the __txn_assert_after_reads holes,
WT-18345); the implementation needs to handle that ordering deliberately. - Deferring the pick-up instead of asserting (as test/format/follower.c already does) remains an alternative if the assertion proves disruptive — the panic-versus-defer question was previously raised in WT-17968 and SERVER-123308.
- is related to
-
WT-17968 Disaggregated storage checkpoint pick-up pinned-timestamp panic check reads unpopulated metadata (dead code)
-
- Blocked
-
-
WT-17969 Follower layered cursor aborts in __clayered_reopen_stable when a read-timestamped reader's parked stable key vanishes from a legitimately-adopted newer checkpoint
-
- Closed
-
-
WT-18339 next/prev on a cursor left positioned on the stable table aborts the node after a step-down when the transaction cannot see the positioned key
-
- Closed
-
-
WT-16467 WiredTiger follower should panic if installing a checkpoint that doesn't cover the pinned timestamp
-
- Closed
-
-
WT-18345 Disaggregated: add defensive checks for the no-overlap invariant timestamped follower readers depend on
-
- Closed
-
- related to
-
WT-17968 Disaggregated storage checkpoint pick-up pinned-timestamp panic check reads unpopulated metadata (dead code)
-
- Blocked
-
-
WT-17969 Follower layered cursor aborts in __clayered_reopen_stable when a read-timestamped reader's parked stable key vanishes from a legitimately-adopted newer checkpoint
-
- Closed
-
-
WT-18339 next/prev on a cursor left positioned on the stable table aborts the node after a step-down when the transaction cannot see the positioned key
-
- Closed
-
-
WT-16467 WiredTiger follower should panic if installing a checkpoint that doesn't cover the pinned timestamp
-
- Closed
-