ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1024
    • Affects Version/s: None
    • Component/s: None
    • None
    • Workload Resilience
    • Fully Compatible
    • ALL
    • Workload Resilience 2026-10-12
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem statement

      The primary-side flow-control standby lag is computed as the wall-time difference between the
      primary's and the standby's last-applied oplog entries
      (DisaggFlowControlShim::_computeLags). Both are ENTRY wall times, so the signal measures
      "how old is the newest entry the standby applied relative to the newest entry written" — which
      counts two things that are not standby lag:

      1. Post-idle write gaps. While the oplog is idle the lag correctly reads 0 (both positions
        frozen at the same entry). The moment the first write after a pause lands, the primary's
        side jumps to now while the standby's side still names the pre-pause entry — the entire
        pause length reads as standby lag until the standby applies the new entry AND a heartbeat
        reports it (apply latency + up to the 2s heartbeat cadence).
      2. Heartbeat reporting cadence under trickle writes. With ~1 write/sec, the lag ratchets up
        ~1s per second between heartbeat reports and snaps to 0 on each report — a permanent 0-3s
        sawtooth on a fully caught-up standby.

      Confirmed in FTDC

      high_cpu_degradation_lightweight_heavyweight_locust, slow-paged variant, primary mongod.0
      (sys-perf patch 6a8163bd, task green — numbers below are from a healthy run;
      task,
      FTDC in its artifacts; timestamps below are UTC in
      high_cpu_degradation_lightweight_heavyweight_locust/mongod.0/metrics.2026-08-16T07-47-25Z-00000):

      • 08:00:09-08:00:14: zero writes, effectiveStandbyLagMs = 0 (standby fully caught up).
      • 08:00:15: one write lands -> effectiveStandbyLagMs jumps to 7793ms (= the elapsed gap),
        holds for 3 samples, then collapses on the next heartbeat report.
      • Query-only/trickle phases show the continuous sawtooth: eff cycles 0 -> 1000 -> 2000 ->
        3000 -> 0 at 1-write/sec with 2s heartbeats.
      • Amplification: _smoothStandbyLag adopts upward jumps instantly but decays at only 0.3/tick,
        so a one-tick 7.8s spike stays above the 1.2s engage threshold for ~6 refreshes.
      • Blast radius in this task: standbyLagDominant flips on 195/982 samples (~20% of the task)
        purely from this artifact (effectiveMatLagMs was 0 throughout); 72 samples read above the
        engage threshold. isLaggedCount delta = 0 — no actual throttling occurred, because
        FlowControl's few-ops isHealthy escape (approximateOpsBetween == -1 when fewer than ~one
        sample bucket separates the positions) held for the whole task.

      Scope

      Checked for the same signatures on the main perf variant (disagg-m8g-perf-11-node):
      bulk_insert_locust,
      bulk_insert_locust_w1,
      find_one_and_update_retry_writes_false_embedded_locust,
      mixed_workloads_locust, and
      ycsb.out_of_cache.95read5update.
      The bulk-insert workloads show only a bounded residue —
      after 73-145s quiet stretches the burst-start lag reads 0.06-1.6s (background trickle writes keep
      the standby's reported entry fresh, so the reading is bounded by trickle spacing + the 2s
      heartbeat cadence, not the pause length), and no trickle sawtooth exceeds the threshold; find_one_and_update (embedded) and
      mixed_workloads never exceed the threshold (max readings 0.88s / 0.21s, with the same
      sub-threshold dominance flapping — standby-dominant on 74% / 79% of samples against zero mat
      lag). Read-heavy phases reproduce it above the
      threshold on the fast variant too: ycsb.out_of_cache.95read5update shows a 5000ms reading (2
      samples, zero writes in the preceding 6s, no throttling) at the read phase's start. The full
      artifact — gap-sized multi-second spikes plus a sustained sawtooth — needs genuinely-zero-write
      gaps plus a slow-to-report standby (the slow-paged variant); severity tracks the write-silence
      duration before the next write, with standby report latency as the amplifier.

      Impact

      • The few-ops escape stops protecting exactly when a burst follows a pause (>~1000 ops in the
        first period): flow control then engages with a lag reading equal to the pause length
        rather than any real backlog, and the decay exponent scales with it — several seconds of
        deep over-throttle at the start of bursts that follow pauses. Bursty workloads (locust
        stage boundaries, index builds after loads) hit this pattern structurally.
      • Dominance flapping: the artifact flips standby-lag dominance on an idle system, switching
        which leg's sustainer count and timestamps FlowControl consumes (observed at ~20% of
        samples above).
      • Telemetry pollution: effectiveStandbyLagMs / smoothedStandbyLagMs in serverStatus/FTDC read
        seconds of "lag" on a caught-up standby, which misleads exactly the investigations that
        rely on them (this defect was found during one).

      Prior art

      The standby-side metric already solves this exact problem:
      ReplicationCoordinatorDisaggregatedStorage::getOplogLagMillis() floors the standby's
      applied wall time with getEffectiveStandbyAppliedWallTime() — the primary's gossiped
      current time from the last heartbeat at which the standby was fully caught up — with a comment
      naming the phenomenon ("suppress false oplog lag spikes on an idle system... interval offset
      between the default heartbeat interval (2s) and the default no-op interval (10s)"). The
      primary-side flow-control shim has no equivalent; the analogous information is available to it
      (it already classifies caught-up refreshes for the rate measurement's StandbyProgress).

            Assignee:
            Haiyin Gu
            Reporter:
            Daotang Yang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: