-
Type:
Bug
-
Resolution: Won't Fix
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Replication
-
None
-
Replication
-
ALL
-
Repl 2026-10-12
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
ReplSetTest.checkReplicaSet() takes an fsyncLock on the primary and holds it for the duration of the consistency checker (checkOplogs / checkReplicatedDataHashes / checkPreImageCollection). The fsyncLock holds MultiDocumentTransactionsBarrier S + ReplicationStateTransition IX + Global S.
If the primary attempts a replication state transition while that lock is held, the transition needs the RSTL in X, cannot get it, and after fassertOnLockTimeoutForStepUpDown seconds the node calls abort().
checkReplicaSet guards against secondaries starting an election:
// Prevent an election, which could start, then hang due to the fsyncLock. jsTestLog(`Freezing nodes: [${secondaries.map((n) => n.host)}]`); secondaries.forEach((secondary) => this.freeze(secondary));
but nothing prevents the primary itself from stepping down because it cannot see a majority. That path is TopologyCoordinator::checkMemberTimeouts(), which is liveness-driven and independent of any secondary election.
Observed failure
jstests/replsets/atlas_initialsync_workflows.js on rhel8-debug-aubsan-experimental, v9.0-staging burn_in. The test partitions a 4-member set, then runs the consistency checks after reconnecting.
21:56:09.65 d20750 I REPL 21216 Member is now in state DOWN (:20741) <- test disconnected it
21:56:09.75 d20750 I REPL 21216 Member is now in state DOWN (:20756) <- test stopped it
21:56:12.791 d20750 I COMMAND 20461 CMD fsync {lock:true} <- checkReplicaSet locks primary
21:56:13.072 d20750 I REPL 21809 Can't see a majority of the set, relinquishing primary
21:56:13.072 d20750 I REPL 21475 Stepping down from primary in response to heartbeat
21:56:15.21 d20750 I REPL 21215 Member is in new state (:20756) <- members already back
21:56:43.073 d20750 F REPL 5675600 Time out exceeded waiting for RSTL, stepUp/stepDown is not
possible thus calling abort()
timeAcquiringMicros W: 30000817
The window between taking the lock and the stepdown decision was 281 ms. The lock manager dump at abort time shows the fsyncLock thread holding RSTL IX, the stepdown thread pending on X, and six further IX requests queued behind it – those are the checker's own dbHash commands, starved by FIFO queueing.
Note the majority loss was already stale by the time it was acted on: both members were reachable again at 21:56:15, ~2.4s after the lock was taken and well before the fassert fired.
Why this is a library bug, not a test bug
The test reconnects its nodes before calling the checks, but reconnectSecondaries() does no settling and the primary's liveness view lags by several heartbeats. Any useBridge test that partitions a set and then calls checkReplicatedDataHashes() has the same exposure. The guard added by SERVER-43226 in 2019 is still structurally intact and still covers only the secondary-election case.
Acceptance Criteria
- checkReplicaSet does not take the fsyncLock while the primary sees fewer than all members as healthy.
- No node that is up and reachable is left unfrozen during the consistency checks.
- jstests/replsets/atlas_initialsync_workflows.js passes under replica_sets_last_lts on an aubsan variant across repeated runs.
- is related to
-
SERVER-43226 Primary takeover during ReplSetTest.stop can hang due to fsyncLock
-
- Closed
-
- related to
-
SERVER-43226 Primary takeover during ReplSetTest.stop can hang due to fsyncLock
-
- Closed
-