primary_casts_vote_on_stepdown.js fails: assert.eq expected term N+1, got N+2

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.0.0-rc1021
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • Fully Compatible
    • ALL
    • v9.0
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      jstests/replsets/primary_casts_vote_on_stepdown.js fails intermittently on debug/TSAN
      variants with:

      uncaught exception: Error: expected 2 to equal NumberLong(3)
      assert.eq@src/mongo/shell/assert.js:310:14
      @jstests/replsets/primary_casts_vote_on_stepdown.js:30:8
      

      Root Cause

      The test steps up node 1 while node 0 is primary in term N, then asserts that node 1's new term is exactly N + 1:

      res = assert.commandWorked(nodes[1].adminCommand("replSetGetStatus"));
      assert.eq(firstPrimaryTerm + 1, res.term);
      

      This assumes the step-up always succeeds in a single election round. In practice, node 0 can independently learn of node 1's higher term via a heartbeat before the replSetRequestVotes command finishes persisting node 0's vote. Node 0's self-initiated stepdown then races with (and interrupts) the in-flight write of the LastVote document, "spoiling" the vote it had just granted (see SERVER-91733 and the "spoiled vote" handling in ReplicationCoordinatorExternalStateImpl::storeLocalLastVoteDocument). The interrupted vote causes node 1's election for term N + 1 to fail with
      CommandFailed: Election failed.;
      ReplSetTest.stepUp() transparently retries, so node 1 ultimately becomes primary in term N + 2 instead of N + 1. The strict assert.eq then fails even though the step-up itself succeeded correctly.

      This race is timing-sensitive and much more likely to manifest on slow/instrumented builds (debug + TSAN), which is why it is seen there and not on standard variants.

      Related

      • SERVER-91733 (spoiled-vote-on-stepdown race in storeLocalLastVoteDocument)
      • Failing task: mongodb-mongo-v9.0-staging-enterprise-rhel8-debug-tsan-replica-sets-auth

            Assignee:
            Matt Olma
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: