Audit and document the WT_DHANDLE_OUTDATED lifecycle and its coupling with dhandle->session_inuse

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: DHandles
    • None
    • Storage Engines - Foundations
    • 49.915
    • None
    • None

      Design notes: https://docs.google.com/document/d/1OlzOKzOea3SFemVTKRRe1efuXsUh7x-Z_eVd2iIUUig/edit?tab=t.828188hu34s9 

      Why this needs an audit

      WT_DHANDLE_OUTDATED means: this dhandle's metadata has been superseded, so don't hand it to new users, let current users finish, never write its content back, and retire it promptly. That one bit is read by four subsystems that each interpret it differently, written from five call sites under two very different lock regimes, and never cleared once set.

      • Its meaning is modulated by session_inuse. Several readers use session_inuse == 0 as "nobody can still be about to use this handle", which is not what the counter means — the flag check and the pin are not one atomic step, so a reader can be between them (WT-18532).
      • It governs future opens only; it never fences in-flight users. Existing references keep using the superseded btree until they drain. Sweep's grace period exists because of this, but the marking sites read as though the mark itself retired the handle.
      • Nothing documents the contract. Each reader encodes its own assumptions locally, so we cannot state — let alone test — what the flag guarantees, or tell a redundant condition from a load-bearing one.

      The known holes are individually narrow, but together they suggest the invariant is implicit rather than designed.

      Writers

      Site What it marks Locks held
      __disagg_mark_btrees_readonly_then_step_down (conn_layered.c) READONLY + OUTDATED on every open writable disaggregated btree checkpoint + schema across the transition; handle-list read lock over the walk; per-tree eviction fence around each mark
      Checkpoint pickup — previous checkpoint's handle OUTDATED none beyond the find
      Checkpoint pickup — live stable handle, via sh_file_key OUTDATED none beyond the find
      Checkpoint pickup / startup / reconfigure — shared metadata table OUTDATED none beyond the find

      Step-down marking only WT_DHANDLE_OPEN dhandles is safe: the schema lock is held across the whole transition and every fresh btree open takes it too, so an open either completes before the walk and is marked, or starts after the role flip and is refused for a live stable tree on a follower.

      Readers

      __wt_conn_dhandle_find — handle reuse

      • Concludes: don't hand an outdated handle to a new user — except a stable checkpoint view someone is already using, which stays shared.
      • Reuse requires all of: btree, READONLY, session_inuse > 0, stable checkpoint-view URI.
      • The checkpoint != NULL loop in the same function skips OUTDATED unconditionally, so the nuanced rule applies only to the URI-suffix form disaggregated views use.
      • Holds the handle-list lock for the walk, but not across the gap between the check and the pin.
      • Missing the flag is not self-healing: the reader binds a superseded handle.
      • The primary OUTDATED ↔ session_inuse coupling, and the site behind WT-18532.

      Sweep

      • Concludes: this handle may be retired.
      • In disaggregated mode, outdated handles are the only thing swept below the handle-count floor.
      • Closes immediately when session_inuse == 0; the grace period covers only .wt_stable/ checkpoint views, which the prune walk and spanning readers still reference.
      • Missing the flag is self-healing — it is never cleared, so a later pass sees it.
      • Also a writer of the same flags word in the expire/close path (WT-18452 / WT-18334).

      Eviction and splits — __wt_btree_is_outdated_disagg

      • Concludes: do not persist this page — return EBUSY, or discard it without writing.
      • Combines DISAGGREGATED, READONLY and OUTDATED; the combinator between the first two is wrong today and is being fixed in WT-18534. OUTDATED is a mandatory conjunct either way.
      • Fenced against the step-down writer by the per-tree eviction lock held across the marking.
      • Not fenced against the pickup writer at all — benign only because the trees pickup marks are READONLY and therefore clean.
      • Missing the flag cannot reach a follower write: reconciling a READONLY tree trips an unconditional assert first, in every build.

      Checkpoint handle gather

      • Concludes: skip this handle when gathering handles for a checkpoint.
      • Always under the checkpoint lock, which both writers also hold — fully serialised, no known race.

      Cursor reopen — the path that deliberately does not read the flag

      • Uses the role-change generation, the checkpoint-generation pin with deferred adoption, and an EBUSY retry when the resolved checkpoint was superseded mid-open.
      • Never consults OUTDATED, though it decides exactly the question the flag exists to answer: whether a bound handle is still current.
      • Least understood of the paths here.

      Known defects

      • WT-18532 (P2, release blocker) — the find/pin gap lets a reader bind a superseded checkpoint after pruning has advanced past it. Reproduced 5/5.
      • WT-18452 / WT-18334 — plain F_SET of the shared flags word races sweep's read-modify-write. Lost OUTDATED is self-healing; a resurrected OPEN/DEAD is crash-class. Never observed directly.
      • AF-19791 — a candidate symptom of the above rather than a separate defect; moving the flag out of the shared word would remove the conflict between the two uses.

      Adjacent and deliberately out of scope: WT_BTREE_READONLY overlaps OUTDATED at nearly every site above and has had its own recent churn — it moved into the atomic flags word, WT-18534 fixes the combinator in the eviction predicate, and WT-18533 is a READONLY/role fence gap. Changes to one flag's rules keep landing on the other's call sites, so the audit should track READONLY's state even though fixing it belongs elsewhere.

      Open questions

      1. Why can't __wt_conn_dhandle_find skip an outdated handle unconditionally? The comment says checkpoint views span eras and sweep keeps them alive for readers and checkpoint tracking — but what breaks if each reader opens its own handle? Presumably the prune accounting, which counts readers through session_inuse on the shared handle, but that's already broken with WT-18532.
      2. Why does checkpoint pickup outdate live btrees? Some part of the investigation on this front was done and posted as a comment to WT-17772.
      3. What does the cursor reopen path actually need? Establish whether working without the flag is by design or by luck, and whether the generation and checkpoint-pin machinery subsumes its role there.
      4. Are there other readers that treat session_inuse == 0 as "no future user"? Known: the find reuse rule, the prune-timestamp advance (WT-18532), sweep's immediate close, and the eviction review gate's EBUSY-vs-discard choice. The first two are unsafe or suspect; the last two need the same scrutiny.

      Scope

      1. Establish and write down what the flag guarantees: who may set it, what each reader is entitled to conclude, and what synchronisation each conclusion depends on — in particular where session_inuse is and is not a valid proxy for "no future user".
      2. Make any changes the audit shows are necessary. Fixes with their own tickets (WT-18532, WT-18452, WT-18334, AF-19791) stay there; this ticket covers what the audit turns up beyond them and closes the open questions.

            Assignee:
            Ivan Kochin
            Reporter:
            Ivan Kochin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: