-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Layered Tables
-
None
-
Storage Engines - Foundations
-
42.738
-
None
-
None
What WiredTiger allows but shouldn't
Nothing bounds a read timestamp from above, and nothing stops a later commit from landing at or below an active read timestamp (the only guard, __txn_assert_after_reads, is diagnostic-only and walks a single connection, so it cannot see a leader's commits from a follower). On a disaggregated node that combination silently breaks snapshot isolation: a timestamped reader migrates to a newer checkpoint view mid-transaction (checkpoint pickup or step-down), where transaction ids are gone and commit_ts <= read_ts is the only visibility filter, so a commit made after the reader's snapshot at or below its read timestamp flips from invisible to visible inside one transaction:
leader: k=w commit @90 (after the follower reader's snapshot) follower: reader at read_timestamp=95 reads k -> WT_NOTFOUND follower: picks up the next checkpoint (not deferred: timestamped readers pin nothing) follower: same reader, same cursor: k -> w <- phantom
Note the pickup protocol's replay-before-adoption rule does not close this: replay guarantees the commit exists in the ingest table before the checkpoint arrives, not that the reader's snapshot covers it. A reader that began before the replay commit correctly ignores the ingest copy (follower-local transaction ids) yet sees the id-less copy in the adopted checkpoint — the reproducer's protocol-faithful variant replays everything before the pickup and still fails.
Why it never happens in MongoDB
MongoDB maintains the invariant itself (the ReadSource::kNoOverlap contract): on a primary, commit timestamps are allocated by a monotone logical clock, so no commit can land below all_durable; on a secondary, commits do land below the (mid-batch overstated) all_durable, but readers are bounded by min(lastApplied, all_durable), which is below every in-flight batch commit. So the break requires a bug or a new read path on the MongoDB side — today WiredTiger would return wrong data with no error.
Proposed strictening (defensive, should never fire)
Encode the invariant WiredTiger already depends on, scoped to disaggregated connections (single-node applications may legally read above all_durable):
- Leader, read side: in __wti_txn_set_read_timestamp, refuse read_ts > all_durable (next to the existing oldest bound). All_durable rather than max committed, so in-flight holes are covered. This alone closes the step-down (survivor reader) variant.
- Leader, commit side (diagnostic-only, deliberately): the other ordering — a commit timestamp allocated below an already-admitted reader — can only come from a monotone-allocation regression on the application side, a bug class diagnostic CI builds catch. The existing __txn_assert_after_reads already asserts it (commit and prepare timestamps above active read timestamps, per-connection — sufficient on the leader, where survivors' readers and writers share the connection). Optionally strengthen the diagnostic to commit_ts > all_durable, which encodes the monotone discipline directly and fires even when no reader is active yet. No production commit path change.
- Follower: WiredTiger cannot compute the bound (the valid one, lastApplied, is a replication-layer value; local all_durable is overstated during out-of-order parallel replay), and a replay commit cannot be refused — so prevention needs either serialized replay (rejected: kills parallel apply) or outside help. WiredTiger can instead prevent the consequence. Preferred: detect-and-invalidate — replay and application readers share the follower connection, so when a commit lands at or below an active reader's read timestamp behind that reader's snapshot, mark the reader's snapshot broken and fail its next operation with WT_ROLLBACK (exact, no deferral, no new API; also replaces the current diagnostic behavior at that spot, a process abort, with a graceful error). Alternatives: the application passes the no-overlap bound (lastApplied) down for a read-side check; or conservative pinning — a reader above the last adopted checkpoint's timestamp pins the checkpoint generation so pickups defer behind it (provably safe via the existing commit-above-stable bound, but over-defers, since legitimate readers normally sit above the last checkpoint).
Document the invariant in the disagg/step-down requirements: read timestamps stay at or below the node's no-overlap point at snapshot time; commit timestamps stay above every active read timestamp.
- related to
-
WT-18259 Investigate reads racing an async step-down against the stable table being marked read-only and outdated
-
- In Progress
-