ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Minor - P4
    • 9.1.0-rc0, 9.1.0-rc1024
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      ReplSetTest.initiateForDisagg() can return before secondaries know who the primary is. Tests that then run a command on a secondary that forwards to the primary fail with PrimarySteppedDown: No primary exists currently. Seen in analyze_shard_key_basic.js on the disagg_override suites.

      Failure

      Task: enterprise-amazon-linux2023-arm64-all-feature-flags-extra-system-deps-sharded-clusters / analyze_shard_key_no_passthrough_disagg_override_2

      Error: command failed: { "ok": 0, "errmsg": "No primary exists currently", "code": 189, "codeName": "PrimarySteppedDown" }
        analyzeShardKey: "testDb.testCollUnsharded", key: { candidateKey0: 1 }  (connection to :20430)
        at testExistingUnshardedCollection (analyze_shard_key_common_tests.js:101)
      

      It happens in the second block of the test, a plain 2-node replica set (_non_multitenant). analyzeShardKey runs on the secondary, which forwards its split-point write to the primary through QueryAnalysisClient::_executeCommandOnPrimaryRemote. That path throws PrimarySteppedDown when getCurrentPrimaryHostAndPort() is empty. The retries in executeCommandOnPrimary run back-to-back with no backoff, so they do not help.

      Root cause

      1. In disagg, node 0 becomes primary about 10s after start (election timeout).
      2. initiateForDisagg() only waits for getPrimary() and awaitSecondaryNodes(). The secondary already reports SECONDARY.
      3. Secondaries learn the primary through heartbeats (2s interval). About 110ms after the step-up (21:18:00.836 to 21:18:00.948) the secondary had not received one.
      4. The test runs analyzeShardKey on the secondary in that window and fails (exit -3).

      The ShardingTest block of the same test passes because ShardingTest calls awaitNodesAgreeOnPrimary itself.

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: