[SERVER-15329] Sync source feedback handshake spins when node is removed Created: 19/Sep/14  Updated: 11/Jul/16  Resolved: 25/Sep/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 2.7.7

Type: Bug Priority: Major - P3
Reporter: Eric Milkie Assignee: Matt Dannenberg
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-15342 clear sync target when they respond w... Closed
Operating System: ALL
Steps To Reproduce:

Run remove1.js

Participants:

 Description   

See remove1.js jstest: while the secondary node is removed but still up, it ends up flooding the primary with network connections in a tight loop. This doesn't happen in 2.6.

 m31001| 2014-09-18T23:53:14.774-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.774-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.774-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.775-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.775-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.775-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.775-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.776-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.776-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.777-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.777-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.777-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.777-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.778-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.778-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.778-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.778-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.779-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.779-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.779-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.779-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.780-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.780-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.780-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.780-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.781-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.781-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.781-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.781-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.782-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.782-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000
 m31001| 2014-09-18T23:53:14.782-0400 I REPLSETS [SyncSourceFeedbackThread] replSet error while handshaking the upstream updater: Node with replica set member ID 1 could not be found in replica set config while attempting to associate it with RID 541ba8965024784187e53514 in replication handshake.  ReplSet Config: { _id: "removeNodes", version: 2, members: [ { _id: 0, host: "build11.nj1.10gen.cc:31000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatTimeoutSecs: 10, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } }
 m31001| 2014-09-18T23:53:14.782-0400 I REPLSETS [SyncSourceFeedbackThread] replset setting syncSourceFeedback to build11.nj1.10gen.cc:31000



 Comments   
Comment by Githook User [ 25/Sep/14 ]

Author:

{u'username': u'dannenberg', u'name': u'matt dannenberg', u'email': u'matt.dannenberg@10gen.com'}

Message: SERVER-15329 add a short sleep if our handshake is rejected because we are not in the current config
Branch: master
https://github.com/mongodb/mongo/commit/1645e89e81ef8446601ce637c5dd3a9b0f124367

Generated at Thu Feb 08 03:37:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.