[SERVER-27166] secondary crashes after being removed from the replica set Created: 23/Nov/16  Updated: 23/Feb/17  Resolved: 23/Nov/16

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sergey Grechin Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-28079 Secondary mongod crashes when removed... Closed
Operating System: ALL
Participants:

 Description   

the ex-secondary crashed about the same time it was removed from the RS configuration.
I'm sure that it is not an intended behaviour (formerly it just entered "OTHER" state).

centos7, mongo 3.2.9

Extract from the mongod.log

2016-11-23T10:44:58.160+0300 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 156007, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007", code: 93 }
2016-11-23T10:44:58.160+0300 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig: Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007, will retry
2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] Cannot find self in new replica set configuration; I must be removed; NodeNotFound: No host described in new configuration 156007 for replica set rs3 maps to this node
2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "rs3", version: 156007, protocolVersion: 1, members: [ { _id: 1, host: "uaDbReplica:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 2, host: "uaDb1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('582643d6ef9bf5817d75fea5') } }
2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] This node is not a member of the config
2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] transition to REMOVED
2016-11-23T10:44:58.183+0300 I NETWORK  [conn41592] end connection 95.183.13.232:43834 (76 connections now open)
2016-11-23T10:44:58.183+0300 I NETWORK  [conn41591] end connection 95.183.13.232:43816 (76 connections now open)
2016-11-23T10:44:58.183+0300 I NETWORK  [conn61491] end connection 185.22.234.231:46820 (76 connections now open)
2016-11-23T10:44:58.183+0300 I NETWORK  [conn61483] end connection 185.22.234.231:45884 (76 connections now open)
2016-11-23T10:44:58.183+0300 I NETWORK  [conn41607] end connection 95.183.13.232:43948 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61485] end connection 185.22.234.231:46214 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61633] end connection 185.22.234.231:48156 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41574] end connection 37.143.15.188:51401 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn62243] end connection 185.22.234.231:51252 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61497] end connection 185.22.234.231:48028 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41622] end connection 95.183.13.232:44058 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41610] end connection 95.183.13.232:43980 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61486] end connection 185.22.234.231:46220 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61363] end connection 185.22.234.231:39466 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41624] end connection 95.183.13.232:44082 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61358] end connection 185.22.234.231:39170 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41648] end connection 95.183.13.232:44176 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61490] end connection 185.22.234.231:46804 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41677] end connection 95.183.13.232:44360 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41667] end connection 95.183.13.232:44296 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn62174] end connection 185.22.234.231:47910 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41594] end connection 95.183.13.232:43856 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn68128] end connection 95.183.13.232:58190 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62176] end connection 185.22.234.231:48068 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61487] end connection 185.22.234.231:46562 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61360] end connection 185.22.234.231:39428 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61492] end connection 185.22.234.231:46838 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41670] end connection 95.183.13.232:44316 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41657] end connection 95.183.13.232:44202 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41662] end connection 95.183.13.232:44244 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41673] end connection 95.183.13.232:44336 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn68576] end connection 95.183.13.232:60564 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41665] end connection 95.183.13.232:44276 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41639] end connection 95.183.13.232:44120 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41672] end connection 95.183.13.232:44334 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62364] end connection 185.22.234.231:42286 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn62246] end connection 185.22.234.231:52566 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41644] end connection 95.183.13.232:44152 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61495] end connection 185.22.234.231:47484 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62365] end connection 185.22.234.231:42314 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62358] end connection 185.22.234.231:41756 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41661] end connection 95.183.13.232:44232 (76 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn41581] end connection 95.183.13.232:43756 (73 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41664] end connection 95.183.13.232:44266 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41647] end connection 95.183.13.232:44164 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61359] end connection 185.22.234.231:39200 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62175] end connection 185.22.234.231:48010 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61496] end connection 185.22.234.231:47774 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41626] end connection 95.183.13.232:44106 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41593] end connection 95.183.13.232:43844 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61488] end connection 185.22.234.231:46564 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41690] end connection 95.183.13.232:44374 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61482] end connection 185.22.234.231:45824 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn61494] end connection 185.22.234.231:47188 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41623] end connection 95.183.13.232:44070 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41609] end connection 95.183.13.232:43970 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41669] end connection 95.183.13.232:44306 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61361] end connection 185.22.234.231:39434 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61489] end connection 185.22.234.231:46566 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn41655] end connection 95.183.13.232:44190 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61362] end connection 185.22.234.231:39464 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61632] end connection 185.22.234.231:48020 (76 connections now open)
2016-11-23T10:44:58.184+0300 I NETWORK  [conn61631] end connection 185.22.234.231:47786 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62362] end connection 185.22.234.231:42278 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn68596] end connection 95.183.13.232:60810 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41642] end connection 95.183.13.232:44142 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41666] end connection 95.183.13.232:44286 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn62361] end connection 185.22.234.231:42250 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn64811] end connection 95.183.13.232:52574 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41658] end connection 95.183.13.232:44212 (76 connections now open)
2016-11-23T10:44:58.185+0300 I NETWORK  [conn41663] end connection 95.183.13.232:44254 (76 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn41606] end connection 95.183.13.232:43936 (75 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn62366] end connection 185.22.234.231:42318 (76 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn62359] end connection 185.22.234.231:42090 (76 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn32383] end connection 185.22.234.231:46922 (73 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn68315] end connection 95.183.13.232:58918 (76 connections now open)
2016-11-23T10:44:58.186+0300 I NETWORK  [conn68393] end connection 95.183.13.232:59654 (75 connections now open)
2016-11-23T10:44:58.186+0300 I -        [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 560
2016-11-23T10:44:58.189+0300 I -        [ReplicationExecutor]
 
***aborting after invariant() failure
 
 
2016-11-23T10:44:58.214+0300 F -        [ReplicationExecutor] Got signal: 6 (Aborted).
 
 0x1322722 0x1321879 0x1322082 0x7fd6c1b82100 0x7fd6c17e65f7 0x7fd6c17e7ce8 0x12a86fb 0xede17a 0xf54a96 0xf0b249 0xf1f4b2 0xf24445 0x1b42e70 0x7fd6c1b7adc5 0x7fd6c18a7ced
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F22722","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F21879"},{"b":"400000","o":"F22082"},{"b":"7FD6C1B73000","o":"F100"},{"b":"7FD6C17B1000","o":"355F7","s":"gsignal"},{"b":"7FD6C17B1000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA86FB","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"ADE17A"},{"b":"400000","o":"B54A96","s":"_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE"},{"b":"400000","o":"B0B249","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb"},{"b":"400000","o":"B1F4B2"},{"b":"400000","o":"B24445","s":"_ZN5mongo4repl19ReplicationExecutor3runEv"},{"b":"400000","o":"1742E70","s":"execute_native_thread_routine"},{"b":"7FD6C1B73000","o":"7DC5"},{"b":"7FD6C17B1000","o":"F6CED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.36.1.el7.x86_64", "version" : "#1 SMP Sun Sep 18 13:04:29 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "6CC0DB94B5F0B88048BD857B35F6D91747E14577" }, { "b" : "7FFF5DCDC000", "elfType" : 3, "buildId" : "C30F6EAE7A15AD672A41176590C84B1BC35C0E02" }, { "b" : "7FD6C2A9B000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7FD6C26B3000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "42AAFD25E9B5F4CE2EFE6309491445B1A92A575D" }, { "b" : "7FD6C24AB000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "1D2AD4EAA62BAD560685A4B8DCCC8D9AA95E22CE" }, { "b" : "7FD6C22A7000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7FD6C1FA5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6DADD94D2A0885D50D09C465EA1970F23FB4629D" }, { "b" : "7FD6C1D8F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7FD6C1B73000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "DF6CCEE00C9F4C983A9464E43D17CC3311B51A8F" }, { "b" : "7FD6C17B1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "53C0918C85FA9CC08D2B57E76467631AB07554AE" }, { "b" : "7FD6C2D08000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F9AF7BA309F063D3BF9657A21436B4DCAC03CF07" }, { "b" : "7FD6C1565000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC2



 Comments   
Comment by Ramon Fernandez Marina [ 23/Nov/16 ]

hq9000, please see the documentation for removing a node from a replica set. The logs above seem to indicate that this node was started with a --replSet argument after being removed.

Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Comment by Sergey Grechin [ 23/Nov/16 ]

it seems now to have problems starting through systemd (systemctl start mongod.service) even after removing the pid file.

I've removed replication section in config - no use.

however starting from command line (mongod -f /etc/mongod.conf) succeeded and I was able to see the data.

Generated at Thu Feb 08 04:14:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.