-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Workload Resilience
-
Fully Compatible
-
ALL
-
Workload Resilience 2026-10-12
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem statement
The primary-side flow-control standby lag is computed as the wall-time difference between the
primary's and the standby's last-applied oplog entries
(DisaggFlowControlShim::_computeLags). Both are ENTRY wall times, so the signal measures
"how old is the newest entry the standby applied relative to the newest entry written" — which
counts two things that are not standby lag:
- Post-idle write gaps. While the oplog is idle the lag correctly reads 0 (both positions
frozen at the same entry). The moment the first write after a pause lands, the primary's
side jumps to now while the standby's side still names the pre-pause entry — the entire
pause length reads as standby lag until the standby applies the new entry AND a heartbeat
reports it (apply latency + up to the 2s heartbeat cadence). - Heartbeat reporting cadence under trickle writes. With ~1 write/sec, the lag ratchets up
~1s per second between heartbeat reports and snaps to 0 on each report — a permanent 0-3s
sawtooth on a fully caught-up standby.
Confirmed in FTDC
high_cpu_degradation_lightweight_heavyweight_locust, slow-paged variant, primary mongod.0
(sys-perf patch 6a8163bd, task green — numbers below are from a healthy run;
task,
FTDC in its artifacts; timestamps below are UTC in
high_cpu_degradation_lightweight_heavyweight_locust/mongod.0/metrics.2026-08-16T07-47-25Z-00000):
- 08:00:09-08:00:14: zero writes, effectiveStandbyLagMs = 0 (standby fully caught up).
- 08:00:15: one write lands -> effectiveStandbyLagMs jumps to 7793ms (= the elapsed gap),
holds for 3 samples, then collapses on the next heartbeat report. - Query-only/trickle phases show the continuous sawtooth: eff cycles 0 -> 1000 -> 2000 ->
3000 -> 0 at 1-write/sec with 2s heartbeats. - Amplification: _smoothStandbyLag adopts upward jumps instantly but decays at only 0.3/tick,
so a one-tick 7.8s spike stays above the 1.2s engage threshold for ~6 refreshes. - Blast radius in this task: standbyLagDominant flips on 195/982 samples (~20% of the task)
purely from this artifact (effectiveMatLagMs was 0 throughout); 72 samples read above the
engage threshold. isLaggedCount delta = 0 — no actual throttling occurred, because
FlowControl's few-ops isHealthy escape (approximateOpsBetween == -1 when fewer than ~one
sample bucket separates the positions) held for the whole task.
Scope
Checked for the same signatures on the main perf variant (disagg-m8g-perf-11-node):
bulk_insert_locust,
bulk_insert_locust_w1,
find_one_and_update_retry_writes_false_embedded_locust,
mixed_workloads_locust, and
ycsb.out_of_cache.95read5update.
The bulk-insert workloads show only a bounded residue —
after 73-145s quiet stretches the burst-start lag reads 0.06-1.6s (background trickle writes keep
the standby's reported entry fresh, so the reading is bounded by trickle spacing + the 2s
heartbeat cadence, not the pause length), and no trickle sawtooth exceeds the threshold; find_one_and_update (embedded) and
mixed_workloads never exceed the threshold (max readings 0.88s / 0.21s, with the same
sub-threshold dominance flapping — standby-dominant on 74% / 79% of samples against zero mat
lag). Read-heavy phases reproduce it above the
threshold on the fast variant too: ycsb.out_of_cache.95read5update shows a 5000ms reading (2
samples, zero writes in the preceding 6s, no throttling) at the read phase's start. The full
artifact — gap-sized multi-second spikes plus a sustained sawtooth — needs genuinely-zero-write
gaps plus a slow-to-report standby (the slow-paged variant); severity tracks the write-silence
duration before the next write, with standby report latency as the amplifier.
Impact
- The few-ops escape stops protecting exactly when a burst follows a pause (>~1000 ops in the
first period): flow control then engages with a lag reading equal to the pause length
rather than any real backlog, and the decay exponent scales with it — several seconds of
deep over-throttle at the start of bursts that follow pauses. Bursty workloads (locust
stage boundaries, index builds after loads) hit this pattern structurally. - Dominance flapping: the artifact flips standby-lag dominance on an idle system, switching
which leg's sustainer count and timestamps FlowControl consumes (observed at ~20% of
samples above). - Telemetry pollution: effectiveStandbyLagMs / smoothedStandbyLagMs in serverStatus/FTDC read
seconds of "lag" on a caught-up standby, which misleads exactly the investigations that
rely on them (this defect was found during one).
Prior art
The standby-side metric already solves this exact problem:
ReplicationCoordinatorDisaggregatedStorage::getOplogLagMillis() floors the standby's
applied wall time with getEffectiveStandbyAppliedWallTime() — the primary's gossiped
current time from the last heartbeat at which the standby was fully caught up — with a comment
naming the phenomenon ("suppress false oplog lag spikes on an idle system... interval offset
between the default heartbeat interval (2s) and the default no-op interval (10s)"). The
primary-side flow-control shim has no equivalent; the analogous information is available to it
(it already classifies caught-up refreshes for the rate measurement's StandbyProgress).
- related to
-
SERVER-136301 Measure disagg flow control standby lag as the age of the oldest unapplied oplog entry
-
- Open
-