Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-87118

Invariant after a forced reconfig with a hybrid index build pending quorum

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 6.0.13, 5.0.24
    • Component/s: None
    • Storage Execution
    • ALL
    • Hide

      1. Create a replica set of 3

      mlaunch init --replicaset --nodes 3

      2. Stop one member
      3. Run in one shell session

      db.getSiblingDB("test").c.insert({a:1})
db.getSiblingDB("test").c.createIndex({b:1})
// This will hang

      4. In another shell session run:

       c = rs.conf()
 c.members[2].priority = 0
 c.members[2].votes = 0
 rs.reconfig(c, {force:true})

      On v5.0.24, 6.0.13 it will crash the server.

      Show
      1. Create a replica set of 3 mlaunch init --replicaset --nodes 3 2. Stop one member 3. Run in one shell session db.getSiblingDB( "test" ).c.insert({a:1}) db.getSiblingDB( "test" ).c.createIndex({b:1}) // This will hang 4. In another shell session run: c = rs.conf() c.members[2].priority = 0 c.members[2].votes = 0 rs.reconfig(c, {force: true }) On v5.0.24, 6.0.13 it will crash the server.

      I have discovered that on v5.0 and 6.0 a foced changing the votes/priorities for a down member while a hybrid index build is in progress crashes the server:

      {"t":{"$date":"2024-02-27T17:24:10.550+11:00"},"s":"F",  "c":"-",        "id":23081,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Invariant failure","attr":{"expr":"status.isA<ErrorCategory::Interruption>() || status.isA<ErrorCategory::ShutdownError>()","msg":"Unexpected error code during index build cleanup: ReadConcernMajorityNotAvailableYet: Committed view disappeared while running operation","file":"src/mongo/db/index_builds_coordinator.cpp","line":2296}}
{"t":{"$date":"2024-02-27T17:24:10.551+11:00"},"s":"F",  "c":"-",        "id":23082,   "ctx":"IndexBuildsCoordinatorMongod-0","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2024-02-27T17:24:10.551+11:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"IndexBuildsCoordinatorMongod-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

      For comparison:

      • v7.0.5 is able to handle a forced reconfig with the index build failed on "ReadConcernMajorityNotAvailableYet: Committed view disappeared while running"
      • v4.4.28 forced reconfig does not seem to be affecting the index build in progress at all. If the primary is stepped down, the new primary will commit the index build successfully

        1. mongod.log
          108 kB
          Dmitry Ryabtsev

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            dmitry.ryabtsev@mongodb.com Dmitry Ryabtsev
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: