Investigate reader latency during disagg step-down / reconfigure-to-follower

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: DHandles
    • None
    • Storage Engines - Foundations
    • 244.661
    • None
    • 8

      Context

      Reads can race with WT stepdown or reconfigure-to-follower. A reader arriving during this window may see elevated latency, in the worst case roughly equal to the stepdown duration itself.

      The contention is with WT stepdown / reconfigure-to-follower. The step-down path runs while holding the checkpoint and schema locks, and its single contention point is __disagg_mark_btrees_readonly_then_step_down, invoked under the dhandle-list read lock, held across the entire walk. For every open disaggregated btree dhandle it takes the eviction-exclusive lock on the tree (waiting for in-flight eviction to drain, so concurrent eviction paths like parent split cannot dirty pages), sets WT_BTREE_READONLY, sets WT_DHANDLE_OUTDATED, and releases the eviction lock. The work is per-dhandle and sequential, so stepdown duration scales with the number of active dhandles and becomes hard to predict.

      In an experiment under high cache pressure with ~60k active dhandles, readers arriving during stepdown saw delays of ~50 ms in the moderately bad case and ~200 ms in the bad case.

      This is not user-visible read unavailability: reads either wait or are rolled back by WT and retried by the server. The concern is latency.

      How readers get pulled in

      1. Readers served from the data-handle cache. The staleness check (__wt_btree_is_outdated_disagg) fires when the btree is disaggregated/readonly and the dhandle is marked outdated. The reader must reacquire a handle, routing it onto the dhandle-list lock, the schema lock (held by stepdown for the whole transition), and the stepdown/eviction synchronization.
      2. Readers opening a genuinely new dhandle. These also contend on the dhandle-list and schema locks, but this is a cold-dhandle path with unavoidable open cost. Measure it, but it is not the focus of this ticket.

      Investigation focus

      For category 1: can we keep cached readers out of the contention with stepdown and eviction entirely?

      What pulls them in is the outdated mark. Per the code, its purpose is to protect a future step-up: a returning leader must open fresh handles rather than reuse resident pages, because carrying them forward lets the drain dirty a page holding an unresolved on-disk prepared cell, a state reconciliation cannot represent. WT_BTREE_READONLY, separately, stops eviction and reconciliation from dirtying pages.

      So the signal does real work, but for a transition that hasn't happened yet, while penalizing readers immediately. Investigate whether it can be delivered without invalidating the handle for readers in the follower era, for example: enforcing the fresh-handle requirement at step-up time (or via a role-era/generation check) instead of a reader-visible flag at step-down; a separate flag that protects eviction/drain while readers continue on the existing readonly handle; reducing or parallelizing the sequential per-tree eviction-lock work; shortening the window stepdown holds the dhandle-list and schema locks.

      Any proposal must preserve correctness around eviction, reconciliation (readonly trees must never reach it), the prepared-cell/drain invariant, schema changes, dhandle lifetime and sweep, and the step-down transition ordering.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Sid Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: