[SERVER-24887] Replica set node allows reconfigurations that are invalid on other versions Created: 04/Jul/16  Updated: 24/Aug/16  Resolved: 28/Jul/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tomas Varaneckas Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to DOCS-8579 Replica Set Upgrade and configuration... Closed
Operating System: ALL
Steps To Reproduce:

1. have a replica set of 3.0.2 + MMAPv1 and 3.2.7 + WT nodes, where 3.0.2 node is primary.
2. set one 3.0.2 node priority > 0 and votes = 0, and make one 3.2.7 node high priority so it would become master upon reconfiguration
3. reconfigure the replicaset on 3.0.2 primary node
4. all 3.2.7 nodes should crash

Participants:

 Description   

We had a replicaset with this configuration (pseudocode):

{
mongo1: { priority: 3, votes: 1, role: PRIMARY, mongo: 3.0.2, storage: MMAPv1 },
mongo2: { priority: 0.5, votes: 1, role: SECONDARY, mongo: 3.0.2, storage: MMAPv1 },
mongo3: { priority: 0.5, votes: 1, role: SECONDARY, mongo: 3.0.2, storage: MMAPV1 },
mongo4: { priority: 0, votes: 1, role: SECONDARY, mongo: 3.2.7, storage: WiredTiger },
mongo5: { priority: 0, votes: 1, role: SECONDARY, mongo: 3.2.7, storage: WiredTiger },
mongo6: { priority: 0, votes: 0, role: SECONDARY, hidden: true, mongo: 3.2.7, storage: WiredTiger }
}

The plan was to switch primary to mongo4 (3.2.7 + WT), so we could upgrade mongo

{1,2,3}, but I managed to crash mongo{4,5,6} with this, on mongo1:

conf = {
   members: [
      { host: mongo1..., priority: 3, votes 1},
      { host: mongo2..., priority: 0.5, votes 1},
      { host: mongo3..., priority: 0.5, votes 0},  // <-- culprit?
      { host: mongo4..., priority: 5, votes 1},
      { host: mongo5..., priority: 0.8, votes 1},
      { host: mongo6..., priority: 0.8, votes 1} // was hidden before
};
rs.reconfig(conf);



In effect, on mongo{4,5,6} mongod crashed, leaving mongo{1,2,3}

in SECONDARY state without being able to elect a PRIMARY.

In logs we found this:

2016-07-04T07:05:31.842+0000 W REPL     [replExecDBWorker-1] Not persisting new configuration in heartbeat response to disk because it is invalid: BadValue: priority must be 0 when non-voting (votes:0)
2016-07-04T07:05:31.842+0000 E REPL     [ReplicationExecutor] Could not validate configuration received from remote node; Removing self until an acceptable configuration arrives; BadValue: priority must be 0 when non-voting (votes:0)
2016-07-04T07:05:31.842+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "repl1", version: 106772, members: [ { _id: 23, host: "mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.8, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 24, host: "mongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.4, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 25, host: "mongo3:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.3, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 26, host: "mongo4:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 27, host: "mongo5:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.6, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 28, host: "mongo6:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }

And then:

2016-07-04T07:05:31.865+0000 I -        [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 560

I'm sure it was a mistake on my end to set priority > 0 and votes = 0 on mongo3, but the way 3.2 + WT nodes reacted was certainly not nice.

Also, please suggest how to perform this switch and upgrade in least dangerous fashion.



 Comments   
Comment by Kelsey Schubert [ 28/Jul/16 ]

Hi spajus,

Thank you for the detailed report. As you correctly identify, MongoDB 3.0 nodes allow replica set reconfigurations that are not valid on later versions of MongoDB. The behavior you observed is expected; during the upgrade process care should be taken to avoid this situation.

If you have a recommendation for how we can improve our documentation around this concern, please open a DOCS ticket.

Thank you,
Thomas

Comment by Tomas Varaneckas [ 04/Jul/16 ]

Cannot give the full output of rs.conf() prior to crash, as it's production cluster and I don't have a way to get it now, but here is what I did:

// on mongo1 (primary)
conf = rs.conf()
conf.members[2].votes = 0 // removed votes on mongo3 to have odd number of voting members, (but priority of this member still > 0)
conf.members[3].priority = 3 // mongo4: set to highest priority in cluster
conf.members[3].votes = 1 // mongo4: votes was 0 before that
conf.members[4].priority = 0.5 // mongo5: priority was 0 before that
conf.members[4].votes = 1 // mongo5: votes was 0 before that
conf.members[5].priority = 0.5 // mongo6: priority was 0 before that
conf.members[5].votes = 1 // mongo6: votes was 0 before that
rs.reconfig(conf)

From logs, this is what conf looked like:

{
  "_id": "repl1",
  "version": 106772,
  "members": [
    {
      "_id": 23,
      "host": "mongo1:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0.8,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 24,
      "host": "mongo2:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0.4,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 25,
      "host": "mongo3:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0.3,
      "tags": {},
      "slaveDelay": 0,
      "votes": 0
    },
    {
      "_id": 26,
      "host": "mongo4:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 3,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 27,
      "host": "mongo5:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0.6,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    },
    {
      "_id": 28,
      "host": "mongo6:27017",
      "arbiterOnly": false,
      "buildIndexes": true,
      "hidden": false,
      "priority": 0.5,
      "tags": {},
      "slaveDelay": 0,
      "votes": 1
    }
  ],
  "settings": {
    "chainingAllowed": true,
    "heartbeatTimeoutSecs": 10,
    "getLastErrorModes": {},
    "getLastErrorDefaults": {
      "w": 1,
      "wtimeout": 0
    }
  }
}

Logs:

On 3.0 + MMAPv1 nodes:

mongo1:

2016-07-04T07:05:31.833+0000 I REPL     [conn14464810] replSetReconfig admin command received from client
2016-07-04T07:05:31.840+0000 I REPL     [conn14464810] replSetReconfig config object with 6 members parses ok
2016-07-04T07:05:31.842+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "repl1", version: 106772, members: [ { _id: 23, host: "mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.8, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 24, host: "mongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.4, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 25, host: "mongo3:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.3, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 26, host: "mongo4:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 27, host: "mongo5:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.6, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 28, host: "mongo6:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-07-04T07:05:31.842+0000 I REPL     [ReplicationExecutor] This node is mongo1:27017 in the config

mongo2:

2016-07-04T07:05:33.321+0000 I NETWORK  [ReplExecNetThread-14007] DBClientCursor::init call() failed
2016-07-04T07:05:33.321+0000 I NETWORK  [ReplExecNetThread-14006] DBClientCursor::init call() failed
2016-07-04T07:05:33.321+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to mongo6:27017; Location10276 DBClientBase::findN: transport error: mongo6:27017 ns: admin.$cmd query: { replSetHeartbeat: "repl1", pv: 1, v: 106771, from: "mongo2:27017", fromId: 24, checkEmpty: false }
2016-07-04T07:05:33.321+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to mongo5:27017; Location10276 DBClientBase::findN: transport error: mongo5:27017 ns: admin.$cmd query: { replSetHeartbeat: "repl1", pv: 1, v: 106771, from: "mongo2:27017", fromId: 24, checkEmpty: false }
2016-07-04T07:05:33.335+0000 I NETWORK  [ReplExecNetThread-14008] DBClientCursor::init call() failed
2016-07-04T07:05:33.335+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to mongo4:27017; Location10276 DBClientBase::findN: transport error: mongo4:27017 ns: admin.$cmd query: { replSetHeartbeat: "repl1", pv: 1, v: 106771, from: "mongo2:27017", fromId: 24, checkEmpty: false }

mongo3:

2016-07-04T07:05:31.836+0000 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 24 whose config version of 106771 doesn't match our config version of 106772", code: 93 }
2016-07-04T07:05:31.838+0000 I NETWORK  [conn12187888] end connection mongo6:57855 (621 connections now open)
2016-07-04T07:05:31.838+0000 I NETWORK  [conn12187695] end connection mongo5:54784 (621 connections now open)
2016-07-04T07:05:31.838+0000 I REPL     [ReplicationExecutor] could not find member to sync from
2016-07-04T07:05:31.913+0000 I NETWORK  [conn12720076] end connection 127.0.0.1:45242 (619 connections now open)
2016-07-04T07:05:33.224+0000 I NETWORK  [initandlisten] connection accepted from mongo2:47182 #12720086 (620 connections now open)
2016-07-04T07:05:33.224+0000 I NETWORK  [conn12720086] end connection mongo2:47182 (619 connections now open)
2016-07-04T07:05:33.829+0000 I NETWORK  [ReplExecNetThread-43360] DBClientCursor::init call() failed
2016-07-04T07:05:33.835+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to mongo4:27017; Location10276 DBClientBase::findN: transport error: mongo4:27017 ns: admin.$cmd query: { replSetHeartbeat: "repl1", pv: 1, v: 106771, from: "mongo3:27017", fromId: 25, checkEmpty: false }

On all 3.2 + WT nodes (mongo4, mongo5, mongo6; identical output):

2016-07-04T07:05:31.846+0000 W REPL     [replExecDBWorker-0] Not persisting new configuration in heartbeat response to disk because it is invalid: BadValue: priority must be 0 when non-voting (votes:0)
2016-07-04T07:05:31.846+0000 E REPL     [ReplicationExecutor] Could not validate configuration received from remote node; Removing self until an acceptable configuration arrives; BadValue: priority must be 0 when non-voting (votes:0)
2016-07-04T07:05:31.846+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "repl1", version: 106772, members: [ { _id: 23, host: "mongo1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.8, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 24, host: "mongo2:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.4, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 25, host: "mongo3:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.3, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 26, host: "mongo4:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 27, host: "mongo5:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.6, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 28, host: "mongo6:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.5, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
2016-07-04T07:05:31.846+0000 I REPL     [ReplicationExecutor] This node is not a member of the config
2016-07-04T07:05:31.846+0000 I REPL     [ReplicationExecutor] transition to REMOVED

Reason of the crash is obvious: mongo 3.0 allowed to set configuration which is invalid in mongo 3.2, so all mongo 3.2 nodes removed themselves, leaving a crippled cluster with 3 members of which only 2 had voting rights.

Comment by Ramon Fernandez Marina [ 04/Jul/16 ]

Can you please provide the full output of rs.conf(), the full configuration document that you used in rs.reconfig(conf), and full logs for all nodes?

Thanks,
Ramón.

Comment by Tomas Varaneckas [ 04/Jul/16 ]

Forgot to mention OS: Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

Generated at Thu Feb 08 04:07:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.