ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Checkpoints
    • None

      Problem

      • __disagg_pick_up_checkpoint (src/conn/conn_layered_checkpoint_pick_up.c:2243) takes WT_WITH_SCHEMA_LOCK around all of __disagg_adopt_checkpoint_meta.
      • That section runs __disagg_merge_checkpoint_meta: a full merge walk of the shared metadata checkpoint against local metadata for every layered table, four cursor pairs, with page-log round trips for shared metadata pages that are not cached. Only the local metadata writes and the shared metadata queue prune need the lock (per the comment at the call site: "The schema lock protects the shared metadata queue from concurrent publish operations").
      • On a freshly restarted follower with 13k - 18k tables one pickup takes 1.7 - 2.7 s while checkpoints arrive every 1.1 s. Pickups run back to back and the schema lock is held ~94% of wall time.
      • Every replicated DDL op on the follower (create, startIndexBuild, dropIdent) waits one full pickup cycle for the lock; the oplog applier runs at <50% speed and lag grows ~0.55 s/s until the lag watchdog or a test deadline fires.

      Evidence (BF-46851, 7 runs on 2026-10-20)

      • Stack dump at the failure: Disagg-3 in __disagg_pick_up_checkpoint -> __disagg_adopt_checkpoint_meta -> __disagg_check_shared_only_table -> __disagg_queued_create_remove -> __wt_disagg_table_latest_create_remove.
      • Waiters release at the pickup boundary: slow applied DDL ends median 2.19 s after the in-progress pickup started, 0.13 s before the next pickup starts; schemaLock wait equals op duration.
      • Same node, same ~16k tables: warm standby 1.1 s pickup cadence and 0.5 s median schema-lock waits; after a restart 2.2 - 2.3 s cadence and 2.1 s waits. The restart, not the table count, pushes the pickup past the checkpoint interval.
      • Per run: restart -> tables -> cadence -> median schema-lock wait: 16099 -> 2.27 s -> 2.1 s; 15781 -> 2.24 s -> 2.0 s; 13137 -> 1.68 s -> 1.6 s; 13111 -> 1.81 s -> 1.7 s; 18268 -> 2.73 s -> 2.6 s; 14228 -> 1.99 s -> 1.9 s; 14901 -> 2.25 s -> 2.2 s.

      Proposed change

      • Split __disagg_adopt_checkpoint_meta into a read phase and an apply phase.
      • Read phase, no schema lock: walk the shared metadata checkpoint, compare against local metadata, build the list of entries to insert, update, drop and the list of shared-only tables to check.
      • Apply phase, under the schema lock: validate the plan against the current schema epoch and queue state (re-check tables that changed since the read phase), write local metadata, update last_ckpt_disaggregated_schema_epoch, prune the queue.
      • Alternative if the two-phase split is too invasive: take and release the schema lock per table or per batch of N entries inside the merge walk so DDL can interleave, with a consistency re-check when the lock is reacquired.
      • Secondary: __wt_disagg_table_latest_create_remove is a linear strcmp scan of the shared metadata queue, called once per shared-only table under the queue spinlock. Index the queue by table name or cache the per-table latest create/remove.

      Expected result

      • Schema lock hold per pickup on a restarted follower is bounded by the local metadata write volume, not by the number of tables or by page-log latency.
      • Replicated DDL on a follower during continuous pickups shows schemaLock waits well below the pickup cadence.
      • disagg_pali_chaos_key on enterprise-amazon-linux2023-arm64-disagg-no-feature-flags-extra-system-deps (10-minute chaos phase, 13k - 18k tables) passes the 30 s dbHash catch-up check after a standby restart.
      • Statistic exposing schema lock hold time per pickup (or extend "disagg: pick up checkpoint time (msecs)" with a locked-section counter).

      Not in scope

      • Replacing the full merge walk with a diff (WT-16188, WT-18819, WT-18820). That work reduces the volume processed; this ticket reduces what is processed under the lock. Both are needed: the startup pickup and the first pickup after step-down still fall back to the full walk.

      Related

      • SERVER-135302 (same mechanism, lag watchdog variant), BF-46851, BF-46427
      • WT-16188, WT-18819, WT-18820 (pickup scaling)
      • WT-17307, WT-18887, WT-18605 (follower lag symptoms)
      • WT-18928, WT-18929 (schema rwlock for open/reopen; does not help here, pickup and DDL both take it exclusively)

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Yury Ershov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: