Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18889

Reelection waiting for a dead system to vote

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: 3.0.3
    • Component/s: Replication
    • Labels:
      None
    • ALL

      tl;dr
      Former Primary becomes Secondary and fails reelection waiting indefinitely for dead node to vote when another Secondary node is cleanly shut down, while third node is known to be dead. This renders whole replica set unresponsive while there is a healthy server - one supposed to be the Primary.

      Whole thing
      The system I'm running consists of three servers in a replica set, two active user facing nodes (mongo01 & mongo02) and one hidden node (dbslave01), for backups. All three are voting, mongo01 & mongo02 have priority:1 and hidden:false, while dbslave01 has priority:0 and hidden:true.
      After migrating the whole cluster to MongoDB3.0 the replication statuses are: mongo02 - PRIMARY, mongo01 - SECONDARY. while dbslave01 is still in "not reacheable/healthy" state, because it crashed during initial startup (though that's not the point of this ticket).

      So having that system state the problem is:
      I log in into mongo01 (SECONDARY) and issue a db.shutdownServer() command. That works fine on mongo01 and it goes down. But at the same time mongo02 (PRIMARY) triggers a reelection, and election fails because it thinks it needs to have at least two votes to become primary again, while dbslave01 is unable to vote (being down) and thus one vote is missing.

      That seems like a errorneous behaviour since unreacheable system should obviously be not expected to vote and the only live server should happily reelect itself to keep being primary. Or ideally election would not trigger because of a secondary shutdown if a primary is still live and serving, because even successful election will still cause connections dropping and service interuption.

            Assignee:
            Unassigned Unassigned
            Reporter:
            mobafill Alex Kotenko
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: