Defer co-located mongot lease/replication startup until after replSetInitiate in resmoke fixtures

XMLWordPrintableJSON

    • Query Integration
    • Fully Compatible
    • v9.0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The resmoke ReplicaSetFixture / ShardedClusterFixture launch a co-located mongot per mongod node and call mongot.await_ready() at the end of each node's setup(), which runs before replSetInitiate (and before mongos is stood up). mongot's lease manager replicates directly from the co-located mongod via a change stream, which requires an initiated replica set with a node in PRIMARY/SECONDARY. Because the lease starts during the pre-initiate window, it fails with NotPrimaryOrSecondary and the fixture either hangs (await_ready retry loop until deadline) or crashes (mongot process exits, surfacing as ServerFailure). This is latent today because the with_mongot_extension suites keep featureFlagSearchExtension off. It becomes a live failure the moment the flag is enabled.

      Startup ordering in the fixture:
      * ReplicaSetFixture.setup() fans out node.setup() across a ThreadPoolExecutor.
      * Each node is a MongoDFixture; MongoDFixture.setup() calls launch_mongot() -> mongot.await_ready() at the end of its own setup (standalone.py:265-266).
      * replSetInitiate / _await_primary() only run after that thread-pool block completes.

      So mongot's lease manager begins replicating while the mongod is still in STARTUP/STARTUP2 and is not yet a valid replication source, yielding NotPrimaryOrSecondary.

      The fixture-side mongot.await_ready() only issues hello (mongot.py:162), so it does not detect the failed lease directly; the symptom surfaces as a hang (0.1s retry loop until the await-ready deadline) or, if mongot exits, as a ServerFailure from the mongot.poll() check.

      With featureFlagSearchExtension off, the mongod<->mongot lease/handshake is effectively established lazily (on first search activity), by which point the replica set is initiated, so mongot tolerating an un-initiated mongod at startup is not fatal. Enabling the search extension brings the lease/connection up eagerly at startup, moving lease acquisition into the exact pre-replSetInitiate window and turning the latent race into a hard failure. This is the same mongot binary and the same ordering bug -- only the timing of the first lease attempt changes.

      Proposed fix:
      Reorder the fixture so mongot's replication-dependent startup happens after the topology is ready to be replicated from:
      * Do not block on mongot readiness inside MongoDFixture.setup(). Split mongot launch (spawn the process) from mongot "await ready" (wait for a working lease/replication).
      * In ReplicaSetFixture.setup(), invoke the deferred mongot.await_ready() only after replSetInitiate + _await_primary() (and, for auto-bootstrap, after the primary is writable). Reuse the existing _defer_mongot_extension plumbing where possible.
      * In ShardedClusterFixture.setup(), defer per-shard mongot readiness until each shard's replica set is initiated and mongos is up.
      * Consider strengthening MongoTFixture.await_ready() to verify the lease/replication is actually established (not just a hello ping), so a failed lease is reported as a fixture failure instead of a silent hang.

            Assignee:
            Daniel Segel
            Reporter:
            Daniel Segel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: