[SERVER-27839] Allow for step downs during reconfig in ReplSetTest initiate Created: 27/Jan/17  Updated: 02/Jul/20  Resolved: 27/Feb/17

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.4.3, 3.5.4

Type: Bug Priority: Major - P3
Reporter: Judah Schvimer Assignee: Judah Schvimer
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-20844 Start ReplSetTests faster wrt initial... Closed
related to SERVER-49305 Remove reconfig retries in our tests Closed
is related to SERVER-27810 Guarantee that replicaset is stable w... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.4
Sprint: Repl 2017-03-06
Participants:
Linked BF Score: 0

 Description   

This occurs because replSetReconfig can sometimes cause the primary to step down and the reconfig in ReplSetTest initiate does not account for this. The reconfig helper is a good example of how to handle reconfigs safely.

The one concern here is that post-SERVER-20844 we make the assumption that nodes[0] is primary after initiate is called. We should ensure that this guarantee is still met, even if the primary steps down during the reconfig. In 3.4 and 3.6 we can use stepUp to accomplish this, but in 3.2 (if we backport SERVER-20844 that far) we either have to accept rare BFs or step down primaries repeatedly until node 0 is elected.



 Comments   
Comment by Githook User [ 06/Mar/17 ]

Author:

{u'username': u'judahschvimer', u'name': u'Judah Schvimer', u'email': u'judah@mongodb.com'}

Message: SERVER-27839 Allow for step downs during reconfig in ReplSetTest initiate

(cherry picked from commit a4e1443629b733c7c0fd44dddcd78e884da848bd)
Branch: v3.4
https://github.com/mongodb/mongo/commit/814b2fe1438fa6c2da698c68a9efdf83ef2702ae

Comment by Githook User [ 27/Feb/17 ]

Author:

{u'username': u'judahschvimer', u'name': u'Judah Schvimer', u'email': u'judah@mongodb.com'}

Message: SERVER-27839 Allow for step downs during reconfig in ReplSetTest initiate
Branch: master
https://github.com/mongodb/mongo/commit/a4e1443629b733c7c0fd44dddcd78e884da848bd

Comment by Judah Schvimer [ 01/Feb/17 ]

This only happens on PV0 because this block exists in PV0 to check for a majority on every heartbeat. There is no real reason to change this behavior, so we should just make the testing changes described in this ticket.

Comment by Judah Schvimer [ 01/Feb/17 ]

One concern is that the above failure has very little time between receiving the reconfig and stepping down. It seems worth looking into if that behavior can and should be improved.

Comment by Benety Goh [ 01/Feb/17 ]

The new ReplSetTest.initiate() procedure increases the config from 1 to N members in a single command. If the primary does not hear from the new members soon enough, it will step down.

[js_test:tags] 2017-02-01T08:03:03.787+0000 d20515| 2017-02-01T08:03:03.787+0000 I REPL     [conn2] replSetInitiate config object with 1 members parses ok
...
[js_test:tags] 2017-02-01T08:03:04.873+0000 d20515| 2017-02-01T08:03:04.873+0000 I REPL     [conn2] replSetReconfig config object with 5 members parses ok
...
[js_test:tags] 2017-02-01T08:03:04.898+0000 d20515| 2017-02-01T08:03:04.888+0000 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "tags", version: 2, members: [ { _id: 0, host: "ip-10-230-229-212:20510", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { server: "0", dc: "ny", ny: "1", rack: "ny.rk1" }, slaveDelay: 0, votes: 1 }, { _id: 1, host: "ip-10-230-229-212:20511", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2.0, tags: { server: "1", dc: "ny", ny: "2", rack: "ny.rk1" }, slaveDelay: 0, votes: 1 }, { _id: 2, host: "ip-10-230-229-212:20512", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: { 2: "this", server: "2", dc: "ny", ny: "3", rack: "ny.rk2" }, slaveDelay: 0, votes: 1 }, { _id: 3, host: "ip-10-230-229-212:20513", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { server: "3", dc: "sf", sf: "1", rack: "sf.rk1" }, slaveDelay: 0, votes: 1 }, { _id: 4, host: "ip-10-230-229-212:20514", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: { server: "4", dc: "sf", sf: "2", rack: "sf.rk2" }, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: { 3 or 4: { sf: 1 }, 3 and 4: { sf: 2 }, 2 dc and 3 server: { dc: 2, server: 3 }, 2: { 2: 1 }, 1 and 2: { 2: 1, server: 1 } }, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('589196379165e547c792b93c') } }
[js_test:tags] 2017-02-01T08:03:04.901+0000 d20515| 2017-02-01T08:03:04.888+0000 I REPL     [ReplicationExecutor] This node is ip-10-230-229-212:20510 in the config
[js_test:tags] 2017-02-01T08:03:04.901+0000 d20515| 2017-02-01T08:03:04.889+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to ip-10-230-229-212:20513
[js_test:tags] 2017-02-01T08:03:04.902+0000 d20515| 2017-02-01T08:03:04.889+0000 I REPL     [ReplicationExecutor] Member ip-10-230-229-212:20512 is now in state STARTUP
[js_test:tags] 2017-02-01T08:03:04.902+0000 d20515| 2017-02-01T08:03:04.889+0000 I REPL     [ReplicationExecutor] can't see a majority of the set, relinquishing primary
[js_test:tags] 2017-02-01T08:03:04.902+0000 d20515| 2017-02-01T08:03:04.889+0000 I REPL     [ReplicationExecutor] Stepping down from primary in response to heartbeat
[js_test:tags] 2017-02-01T08:03:04.902+0000 b20513| 2017-02-01T08:03:04.890+0000 I NETWORK  [main] connection accepted from 10.230.229.212:41162 #5 (1 connection now open)
[js_test:tags] 2017-02-01T08:03:04.903+0000 b20510| 2017-02-01T08:03:04.890+0000 I BRIDGE   [thread3] Received an empty response, end connection 10.230.229.212:45408
[js_test:tags] 2017-02-01T08:03:04.903+0000 2017-02-01T08:03:04.891+0000 E QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetReconfig' on host 'ip-10-230-229-212:20510'  :
[js_test:tags] 2017-02-01T08:03:04.903+0000 DB.prototype.runCommand@src/mongo/shell/db.js:132:1
[js_test:tags] 2017-02-01T08:03:04.904+0000 ReplSetTest/this.initiate/<@src/mongo/shell/replsettest.js:751:29
[js_test:tags] 2017-02-01T08:03:04.904+0000 assert.retry@src/mongo/shell/assert.js:229:13
[js_test:tags] 2017-02-01T08:03:04.904+0000 ReplSetTest/this.initiate@src/mongo/shell/replsettest.js:750:1
[js_test:tags] 2017-02-01T08:03:04.904+0000 @jstests/replsets/tags.js:12:1
[js_test:tags] 2017-02-01T08:03:04.904+0000 @jstests/replsets/tags.js:1:2
[js_test:tags] 2017-02-01T08:03:04.905+0000 failed to load: jstests/replsets/tags.js

Comment by Judah Schvimer [ 01/Feb/17 ]

This also looks to mostly be an issue for PV0, so it's possible that there's a timing bug there and we're not rescheduling some liveness checks correctly in PV0.

Comment by Judah Schvimer [ 01/Feb/17 ]

Reconfig will actually fail if the new config makes the current primary unelectable. Reconfig can lead to a step down if nodes are added with poor timing and suddenly the primary can't see a majority of the set when it checks.

Comment by Spencer Brody (Inactive) [ 30/Jan/17 ]

My understanding was reconfig only causes a stepdown if new config makes the current primary unelectable. For the reconfig done during intiation per SERVER-20844, this shouldn't ever be the case.

Generated at Thu Feb 08 04:16:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.