Follower timestamped reader can bind a superseded checkpoint after its data was pruned from the ingest table - stale read

XMLWordPrintableJSON

    • Storage Engines - Foundations
    • 94.45
    • None
    • None

      Problem

      On a disaggregated-storage follower, a timestamped (or non-transactional) reader resolving the newest stable checkpoint is not atomic with the open of that checkpoint's dhandle: the OUTDATED check happens in the dhandle find (_wt_conn_dhandle_find), the session_inuse pin happens later in the open (wt_cursor_dhandle_incr_use), and nothing revalidates after the open. A thread suspended between the two for the duration of a checkpoint pickup opens a checkpoint that the pickup already retired: the merge marked its dhandle OUTDATED (wti_conn_dhandle_outdated) and the prune walk advanced the ingest prune_timestamp past it (_layered_update_ingest_table_prune_timestamp, which has no pinned-timestamp clamp). The reader then reads the old stable checkpoint while the ingest content covering the gap was pruned — a stale read: an old committed value is returned for a key updated between the two checkpoints.

      Note this is not only an interleaving problem but a lack of synchronization: on the dhandle-reuse path there is no lock and no happens-before edge (no lock-free synchronization either) that would guarantee a reader observes OUTDATED, or that the prune walk observes the reader's pin — the protocol currently relies on ordering and timing rather than on any synchronizing primitive. The suspension interleaving is just how we make the missing synchronization observable; the gap exists regardless of it.

      Only timestamped / non-transactional readers are exposed: untimestamped snapshots pin WT_GEN_DISAGG_CKPT and defer the adoption.

      Reproduced

      Deterministic reproducer in a comment below (temporary, not for commit): an env-armed sleep hook in __wt_session_get_dhandle suspends the reader's open after the find matched, before the inuse pin. The race run returns the stale value 5/5; the control run (reader starts after pickup+prune) returns the correct value.

      Impact

      Wrong (stale) read results on followers in an extreme scheduling interleaving. No committed data is lost.

      Fix direction

      Validate-after-open: acquire-load last_checkpoint_meta_lsn after binding the stable dhandle and retry the bind if it advanced past the resolved checkpoint (same pattern as __txn_snapshot_validate_disagg). Alternatively make the check-and-pin atomic via a dedicated lock around find→inuse. Independent of WT-18452's flag-atomicity fix; that fix alone does not close this hole.

      Related

      WT-18452 (unlocked OUTDATED update), WT-17772, WT-17968, WT-18408

        1. wt-18542.patch
          4 kB
          Ivan Kochin

            Assignee:
            Ivan Kochin
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: