Checkpoint fake-path decision can race with concurrent btree->original clear, skipping a dirty tree's reconcile

XMLWordPrintableJSON

    • Storage Engines - Persistence
    • 345.907
    • SE Persistence backlog
    • None

      Background

      WT-16419 fixed a TSan-flagged data race on btree->original: it was cleared with an atomic CAS (__wt_btree_disable_bulk) but read with plain loads elsewhere, which is UB under the C memory model even though the flag only ever transitions 1 -> 0 once. That ticket made all readers use atomic loads (relaxed ordering), which removes the UB/tearing but intentionally does not change behavior.

      The deeper issue

      __checkpoint_tree in src/checkpoint/checkpoint_txn.c decides whether to take the "fake checkpoint" path based solely on btree->original:

      if (is_checkpoint && __wt_atomic_load_uint8_relaxed(&btree->original)) {
          __wt_checkpoint_tree_reconcile_update(session, &ta);
          fake_ckpt = true;
          __wt_checkpoint_update_generation(session, btree);
          goto fake;
      }
      

      If this read observes a stale 1 while the tree has concurrently been dirtied (e.g. by a live_restore worker calling _wti_live_restore_fs_restore_file > _wt_btree_disable_bulk, see src/live_restore/live_restore_fs.c:956), checkpoint takes the fake/empty path and skips the real reconcile entirely – it never even inspects btree>modified. Contrast with the real checkpoint path a few lines later, which clears btree->modified under an explicit WT_FULL_BARRIER() specifically because that flag's ordering is checkpoint-critical; original bypasses that protection.

      Because a checkpoint is a named, immutable point-in-time snapshot (not just a periodic sync), this is not simply "picked up by the next checkpoint" – if this exact checkpoint is later read (e.g. via backup or a checkpoint cursor), it can appear empty despite genuinely dirty, checkpoint-worthy content having existed in the tree in real time.

      No choice of memory order on btree->original alone can close this gap: the live-restore worker clears original first and dirties pages afterward in program order, and there is no lock or acquire/release pairing between checkpoint's decision and that worker's write sequence (per WT-16419's analysis, the live_restore worker runs in its own thread group, does not hold the schema lock, and is not serialised against checkpoint through the dhandle). Establishing "stale original implies no dirty data yet" would require either real mutual exclusion between the two, or having the fake-path decision also consult btree->modified (which does have proper barriers) instead of trusting original as the sole gate.

      Suggested direction

      Have __checkpoint_tree's fake-path decision also check btree->modified (or equivalent) before committing to the fake/empty checkpoint, so a concurrently-dirtied "original" tree cannot be silently skipped. Needs design discussion on the right way to serialize/read that additional state safely.

      Related: WT-16419, WT-17428, WT-18052.

            Assignee:
            Albert Song
            Reporter:
            Chenhao Qu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: