-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Query Integration
-
Fully Compatible
-
v9.0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The resmoke ReplicaSetFixture / ShardedClusterFixture launch a co-located mongot per mongod node and call mongot.await_ready() at the end of each node's setup(), which runs before replSetInitiate (and before mongos is stood up). mongot's lease manager replicates directly from the co-located mongod via a change stream, which requires an initiated replica set with a node in PRIMARY/SECONDARY. Because the lease starts during the pre-initiate window, it fails with NotPrimaryOrSecondary and the fixture either hangs (await_ready retry loop until deadline) or crashes (mongot process exits, surfacing as ServerFailure). This is latent today because the with_mongot_extension suites keep featureFlagSearchExtension off. It becomes a live failure the moment the flag is enabled.
Startup ordering in the fixture:
* ReplicaSetFixture.setup() fans out node.setup() across a ThreadPoolExecutor.
* Each node is a MongoDFixture; MongoDFixture.setup() calls launch_mongot() -> mongot.await_ready() at the end of its own setup (standalone.py:265-266).
* replSetInitiate / _await_primary() only run after that thread-pool block completes.
So mongot's lease manager begins replicating while the mongod is still in STARTUP/STARTUP2 and is not yet a valid replication source, yielding NotPrimaryOrSecondary.
The fixture-side mongot.await_ready() only issues hello (mongot.py:162), so it does not detect the failed lease directly; the symptom surfaces as a hang (0.1s retry loop until the await-ready deadline) or, if mongot exits, as a ServerFailure from the mongot.poll() check.
With featureFlagSearchExtension off, the mongod<->mongot lease/handshake is effectively established lazily (on first search activity), by which point the replica set is initiated, so mongot tolerating an un-initiated mongod at startup is not fatal. Enabling the search extension brings the lease/connection up eagerly at startup, moving lease acquisition into the exact pre-replSetInitiate window and turning the latent race into a hard failure. This is the same mongot binary and the same ordering bug -- only the timing of the first lease attempt changes.
Proposed fix:
Reorder the fixture so mongot's replication-dependent startup happens after the topology is ready to be replicated from:
* Do not block on mongot readiness inside MongoDFixture.setup(). Split mongot launch (spawn the process) from mongot "await ready" (wait for a working lease/replication).
* In ReplicaSetFixture.setup(), invoke the deferred mongot.await_ready() only after replSetInitiate + _await_primary() (and, for auto-bootstrap, after the primary is writable). Reuse the existing _defer_mongot_extension plumbing where possible.
* In ShardedClusterFixture.setup(), defer per-shard mongot readiness until each shard's replica set is initiated and mongos is up.
* Consider strengthening MongoTFixture.await_ready() to verify the lease/replication is actually established (not just a hello ping), so a failed lease is reported as a fixture failure instead of a silent hang.