AdaptiveCheckpointPolicy (SERVER-120876) causes multi-second read blackouts on disagg standbys

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Engines
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Since SERVER-120876 (23590f32c05e, merged Jul 10) landed, disagg standbys suffer 3–5 second read blackouts: whenever a checkpoint follows a long adaptive wait, the standby's pickup of that oversized checkpoint runs under the WT schema lock for seconds (pickup time observed 123ms → 3274–5034ms), blocking all reads. Every failed op in all six analyzed sys-perf tasks is a secondary read dying at its 3s timeout on the standby. The class is invisible to driver-level failover: the standby stays selectable (heartbeats fine), so reads are sent and die.

      The producing primary shows a small, genuinely server-side effect at checkpoint start (~0.5–1 aggregate thread-seconds of schema-lock/dhandle waits) — real but sub-second per op. The dramatic concurrent write-throughput dip (→ ~0 for 2–7s) in these runs is mostly a client-side artifact of the synchronous workload: the 64 sequential mixed-op workers park on their next secondary read during the standby blackout and stop issuing writes. Evidence: writes that completed during the dip carried normal latencies (8–25 ms, never near the 5s wtimeout), execution tickets stayed free, requests stopped arriving (network.numRequests 1679/s → 3/s), and the primary-targeted 3s-timeout reads (FindOne) never failed in any run — unlike the standby-targeted ones.

      The sys-perf suite (disagg-availability-pali.arm.aws) has been failing all 6 mongod-event tasks since Jul 16 (BF-44818). (The harness message says "unexplained write outage" — a label misnomer; the failing ops are secondary reads.)

      Example failing mainline tasks (version 33b7f283c6, Jul 16):

      • pali_graceful_stop_standby_mongod_locust (fails on both executions)
      • Same-version failures with the identical signature: pali_kill_primary_mongod_locust, pali_graceful_stop_primary_mongod_locust, pali_kill_standby_mongod_locust, pali_mongod_planned_shutdown_locust, pali_kill_primary_mid_startup2_locust — while all 13 SLS-process-event tasks are green.
      • FTDC-analyzed red probe: 6a61cb22 / binary d69e6788bb46

      Root cause (FTDC-verified): the adaptive policy stretches the checkpoint interval from a fixed 5s to 20–60s under moderate write load. A disagg checkpoint applies per-table metadata operations under the WT schema lock on the producing primary, and the standby's checkpoint pickup applies the same accumulated metadata under its own schema lock. Both critical sections scale with the accumulated interval: at 5s cadence they are sub-100ms and invisible; at 40–60s the pickup reaches 3–5 seconds and blacks out all reads on the standby.

      The other failures can be found in https://spruce.corp.mongodb.com/version/sys_perf_33b7f283c643385bc0a42dbe040785c9c3e658ba/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

        1. image-2026-07-24-13-21-20-230.png
          31 kB
          Luke Pearson
        2. Screenshot 2026-07-24 at 4.11.41 pm.png
          192 kB
          Alexander Gorrod

            Assignee:
            Unassigned
            Reporter:
            Daotang Yang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: