Disaggregated: add defensive checks for the no-overlap invariant timestamped follower readers depend on

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Layered Tables
    • None
    • Storage Engines - Foundations
    • 43.639
    • None
    • None

      What WiredTiger allows but shouldn't

      Nothing bounds a read timestamp from above, and nothing stops a later commit from landing at or below an active read timestamp (the only guard, __txn_assert_after_reads, is diagnostic-only and walks a single connection, so it cannot see a leader's commits from a follower). On a disaggregated node that combination silently breaks snapshot isolation: a timestamped reader migrates to a newer checkpoint view mid-transaction (checkpoint pickup or step-down), where transaction ids are gone and commit_ts <= read_ts is the only visibility filter, so a commit made after the reader's snapshot at or below its read timestamp flips from invisible to visible inside one transaction:

      leader:   k=w commit @90            (after the follower reader's snapshot)
      follower: reader at read_timestamp=95 reads k -> WT_NOTFOUND
      follower: picks up the next checkpoint (not deferred: timestamped readers pin nothing)
      follower: same reader, same cursor: k -> w      <- phantom
      

      Note the pickup protocol's replay-before-adoption rule does not close this: replay guarantees the commit exists in the ingest table before the checkpoint arrives, not that the reader's snapshot covers it. A reader that began before the replay commit correctly ignores the ingest copy (follower-local transaction ids) yet sees the id-less copy in the adopted checkpoint — the reproducer's protocol-faithful variant replays everything before the pickup and still fails.

      Why it never happens in MongoDB

      MongoDB maintains the invariant itself (the ReadSource::kNoOverlap contract): on a primary, commit timestamps are allocated by a monotone logical clock, so no commit can land below all_durable; on a secondary, commits do land below the (mid-batch overstated) all_durable, but readers are bounded by min(lastApplied, all_durable), which is below every in-flight batch commit. So the break requires a bug or a new read path on the MongoDB side — today WiredTiger would return wrong data with no error.

      Proposed strictening (defensive, should never fire)

      Encode the invariant WiredTiger already depends on, scoped to disaggregated connections (single-node applications may legally read above all_durable):

      • Leader, read side: in __wti_txn_set_read_timestamp, refuse read_ts > all_durable (next to the existing oldest bound). All_durable rather than max committed, so in-flight holes are covered. This alone closes the step-down (survivor reader) variant.
      • Leader, commit side (diagnostic-only, deliberately): the other ordering — a commit timestamp allocated below an already-admitted reader — can only come from a monotone-allocation regression on the application side, a bug class diagnostic CI builds catch. The existing __txn_assert_after_reads already asserts it (commit and prepare timestamps above active read timestamps, per-connection — sufficient on the leader, where survivors' readers and writers share the connection). Optionally strengthen the diagnostic to commit_ts > all_durable, which encodes the monotone discipline directly and fires even when no reader is active yet. No production commit path change.
      • Follower: WiredTiger cannot compute the bound (the valid one, lastApplied, is a replication-layer value; local all_durable is overstated during out-of-order parallel replay), and a replay commit cannot be refused — so prevention needs either serialized replay (rejected: kills parallel apply) or outside help. WiredTiger can instead prevent the consequence. Preferred: detect-and-invalidate — replay and application readers share the follower connection, so when a commit lands at or below an active reader's read timestamp behind that reader's snapshot, mark the reader's snapshot broken and fail its next operation with WT_ROLLBACK (exact, no deferral, no new API; also replaces the current diagnostic behavior at that spot, a process abort, with a graceful error). Alternatives: the application passes the no-overlap bound (lastApplied) down for a read-side check; or conservative pinning — a reader above the last adopted checkpoint's timestamp pins the checkpoint generation so pickups defer behind it (provably safe via the existing commit-above-stable bound, but over-defers, since legitimate readers normally sit above the last checkpoint).

      Document the invariant in the disagg/step-down requirements: read timestamps stay at or below the node's no-overlap point at snapshot time; commit timestamps stay above every active read timestamp.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: