Data loss when a picked up checkpoint's oldest timestamp > current read timestamp

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Layered Tables
    • Storage Engines - Foundations
    • 9.207
    • None
    • None

      No layer prevents a follower from adopting stable content whose oldest timestamp exceeds the read timestamp of an active transaction. The server explicitly permits readers behind the oldest timestamp and accepts a WiredTiger-side crash as the detection mechanism, on the grounds that the 30-minute history window makes the overlap rare (SERVER-118879). WiredTiger's side of that agreement, the WT-16467 pick-up panic, is currently dead code: the comparison in {}wt_disagg_pick_up_checkpoint _reads metadata.oldest_timestamp before }}{{_wt_disagg_parse_meta populates it (WT-17968), and the attempt to enable it is blocked on SERVER-122575 because the pinned-timestamp comparison false-positives on a stale local oldest timestamp during the rollout window.

      When the overlap happens, the reader's data is unrecoverable by construction: the leader's reconciliation discards a key and its history once its tombstone is globally visible at the leader's oldest timestamp, so the new checkpoint contains neither the key nor a history store version serveable at the reader's timestamp. The consequences on the follower:

      • An unpositioned timestamped reader silently loses the key: __clayered_can_advance_stable takes the read-timestamp fast path, reopens the stable cursor onto the new checkpoint, and a subsequent search returns WT_NOTFOUND for a key the same transaction read earlier. Reproducer: test_layered_pinned_read_adopted_oldest.py (fails on the current develop tip).
      • A positioned reader is protected only by the guard WT-17969 hoisted above the fast path (see the FIXME-WT-17968 comment there), and only on the checkpoint pick-up path; the same condition reached through the role-change reopen (step-down) aborts on "upgrading a positioned stable cursor" (WT-18339).

      Proposed solution

      1. Assert at stable-content adoption (checkpoint pick-up and role-change) that the incoming content's oldest timestamp does not exceed the read timestamp of any active transaction — the oldest is already in the adopted checkpoint metadata; the reader side is a scan of the active transactions' read timestamps. Deliberately not the pinned timestamp: pinned includes the connection's local oldest, which lags during the SERVER-122575 rollout gap and produced the false positives that got the WT-17968 panic reverted (along with the capped-pinned bug, draft PR #14149). A reader-only minimum tests exactly the unsafe condition — readers are a term of pinned, so it catches every case the pinned-based panic would — with no dependency on the server-side rollout, and it also covers the role-change adoption path the pick-up panic never did. This effectively supersedes the WT-17968 enablement path.
      2. With that invariant enforced, move the positioned-on-stable check in {}clayered_can_advance_stable _back inside the no-read-timestamp branch, as its FIXME anticipates. Under oldest <= read timestamp the move is safe: a key visible at the reader's timestamp cannot have its removal globally visible at the incoming oldest, so the key (or its history store version) survives and the position transfer in }}{{_clayered_reopen_stable succeeds; the reopen assertion becomes unreachable for timestamped readers.

      Notes:

      • The check races against a transaction publishing its read timestamp concurrently with the adoption-time scan (the same ordering family as the __txn_assert_after_reads holes, WT-18345); the implementation needs to handle that ordering deliberately.
      • Deferring the pick-up instead of asserting (as test/format/follower.c already does) remains an alternative if the assertion proves disruptive — the panic-versus-defer question was previously raised in WT-17968 and SERVER-123308.

            Assignee:
            Ivan Kochin
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: