Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-72774

A node in quiesce mode can win election

    XMLWordPrintableJSON

Details

    • Replication
    • Fully Compatible
    • ALL
    • v6.3, v6.2, v6.0, v5.0, v4.4
    • Repl 2023-02-06, Repl 2023-02-20

    Description

      A replica set member being in quiesce mode can end up winning an election:

      {"t":{"$date":"2023-01-04T02:45:08.421+00:00"},"s":"I",  "c":"COMMAND",  "id":4695400, "ctx":"conn67","msg":"Terminating via shutdown command","attr":{"force":true,"timeoutSecs":15}}
      {"t":{"$date":"2023-01-04T02:45:08.421+00:00"},"s":"I",  "c":"REPL",     "id":4794602, "ctx":"conn67","msg":"Attempting to enter quiesce mode"}
      {"t":{"$date":"2023-01-04T02:45:08.421+00:00"},"s":"I",  "c":"REPL",     "id":4695102, "ctx":"conn67","msg":"Entering quiesce mode for shutdown","attr":{"quiesceTimeMillis":14999}}
       
      {"t":{"$date":"2023-01-04T02:45:12.066+00:00"},"s":"I",  "c":"ELECTION", "id":21450,   "ctx":"ReplCoord-0","msg":"Election succeeded, assuming primary role","attr":{"term":158}}
      {"t":{"$date":"2023-01-04T02:45:12.066+00:00"},"s":"I",  "c":"REPL",     "id":21358,   "ctx":"ReplCoord-0","msg":"Replica set state transition","attr":{"newState":"PRIMARY","oldState":"SECONDARY"}}
      

      However, being in that mode will not allow the peers to sync from the newly elected primary:

      {"t":{"$date":"2023-01-04T02:45:12.978+00:00"},"s":"I",  "c":"REPL",     "id":3873117, "ctx":"BackgroundSync","msg":"Choosing primary as sync source","attr":{"primary":"atlas-mbh24j-shard-00-01.nt8am.mongodb.net:27017"}}
      {"t":{"$date":"2023-01-04T02:45:12.978+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplCoordExternNetwork","msg":"Connecting","attr":{"hostAndPort":"atlas-mbh24j-shard-00-01.nt8am.mongodb.net:27017"}}
      {"t":{"$date":"2023-01-04T02:45:12.996+00:00"},"s":"I",  "c":"REPL",     "id":5579707, "ctx":"ReplCoordExtern-0","msg":"Denylisting candidate due to error","attr":{"candidate":"atlas-mbh24j-shard-00-01.nt8am.mongodb.net:27017","error":{"code":91,"codeName":"ShutdownInProgress","errmsg":"The server is in quiesce mode and will shut down","remainingQuiesceTimeMillis":10428},"denylistDurationSeconds":10,"denylistUntil":{"$date":"2023-01-04T02:45:22.996Z"}}}
      

      This can lead to oplog divergion and, later, a rollback when the shutdown node is restarted to rejoin the replica set.

      Attachments

        Activity

          People

            wenbin.zhu@mongodb.com Wenbin Zhu
            dmitry.ryabtsev@mongodb.com Dmitry Ryabtsev
            Votes:
            0 Vote for this issue
            Watchers:
            23 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: