-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Admin
-
None
-
ALL
the ex-secondary crashed about the same time it was removed from the RS configuration.
I'm sure that it is not an intended behaviour (formerly it just entered "OTHER" state).
centos7, mongo 3.2.9
Extract from the mongod.log
2016-11-23T10:44:58.160+0300 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 156007, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007", code: 93 } 2016-11-23T10:44:58.160+0300 I REPL [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig: Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007, will retry 2016-11-23T10:44:58.182+0300 I REPL [ReplicationExecutor] Cannot find self in new replica set configuration; I must be removed; NodeNotFound: No host described in new configuration 156007 for replica set rs3 maps to this node 2016-11-23T10:44:58.182+0300 I REPL [ReplicationExecutor] New replica set config in use: { _id: "rs3", version: 156007, protocolVersion: 1, members: [ { _id: 1, host: "uaDbReplica:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 2, host: "uaDb1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('582643d6ef9bf5817d75fea5') } } 2016-11-23T10:44:58.182+0300 I REPL [ReplicationExecutor] This node is not a member of the config 2016-11-23T10:44:58.182+0300 I REPL [ReplicationExecutor] transition to REMOVED 2016-11-23T10:44:58.183+0300 I NETWORK [conn41592] end connection 95.183.13.232:43834 (76 connections now open) 2016-11-23T10:44:58.183+0300 I NETWORK [conn41591] end connection 95.183.13.232:43816 (76 connections now open) 2016-11-23T10:44:58.183+0300 I NETWORK [conn61491] end connection 185.22.234.231:46820 (76 connections now open) 2016-11-23T10:44:58.183+0300 I NETWORK [conn61483] end connection 185.22.234.231:45884 (76 connections now open) 2016-11-23T10:44:58.183+0300 I NETWORK [conn41607] end connection 95.183.13.232:43948 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61485] end connection 185.22.234.231:46214 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61633] end connection 185.22.234.231:48156 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41574] end connection 37.143.15.188:51401 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn62243] end connection 185.22.234.231:51252 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61497] end connection 185.22.234.231:48028 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41622] end connection 95.183.13.232:44058 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41610] end connection 95.183.13.232:43980 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61486] end connection 185.22.234.231:46220 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61363] end connection 185.22.234.231:39466 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41624] end connection 95.183.13.232:44082 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61358] end connection 185.22.234.231:39170 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41648] end connection 95.183.13.232:44176 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61490] end connection 185.22.234.231:46804 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41677] end connection 95.183.13.232:44360 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41667] end connection 95.183.13.232:44296 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn62174] end connection 185.22.234.231:47910 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41594] end connection 95.183.13.232:43856 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn68128] end connection 95.183.13.232:58190 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62176] end connection 185.22.234.231:48068 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61487] end connection 185.22.234.231:46562 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61360] end connection 185.22.234.231:39428 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61492] end connection 185.22.234.231:46838 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41670] end connection 95.183.13.232:44316 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41657] end connection 95.183.13.232:44202 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41662] end connection 95.183.13.232:44244 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41673] end connection 95.183.13.232:44336 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn68576] end connection 95.183.13.232:60564 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41665] end connection 95.183.13.232:44276 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41639] end connection 95.183.13.232:44120 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41672] end connection 95.183.13.232:44334 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62364] end connection 185.22.234.231:42286 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn62246] end connection 185.22.234.231:52566 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41644] end connection 95.183.13.232:44152 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61495] end connection 185.22.234.231:47484 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62365] end connection 185.22.234.231:42314 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62358] end connection 185.22.234.231:41756 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41661] end connection 95.183.13.232:44232 (76 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn41581] end connection 95.183.13.232:43756 (73 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41664] end connection 95.183.13.232:44266 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41647] end connection 95.183.13.232:44164 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61359] end connection 185.22.234.231:39200 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62175] end connection 185.22.234.231:48010 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61496] end connection 185.22.234.231:47774 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41626] end connection 95.183.13.232:44106 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41593] end connection 95.183.13.232:43844 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61488] end connection 185.22.234.231:46564 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41690] end connection 95.183.13.232:44374 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61482] end connection 185.22.234.231:45824 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn61494] end connection 185.22.234.231:47188 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41623] end connection 95.183.13.232:44070 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41609] end connection 95.183.13.232:43970 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41669] end connection 95.183.13.232:44306 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61361] end connection 185.22.234.231:39434 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61489] end connection 185.22.234.231:46566 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn41655] end connection 95.183.13.232:44190 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61362] end connection 185.22.234.231:39464 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61632] end connection 185.22.234.231:48020 (76 connections now open) 2016-11-23T10:44:58.184+0300 I NETWORK [conn61631] end connection 185.22.234.231:47786 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62362] end connection 185.22.234.231:42278 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn68596] end connection 95.183.13.232:60810 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41642] end connection 95.183.13.232:44142 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41666] end connection 95.183.13.232:44286 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn62361] end connection 185.22.234.231:42250 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn64811] end connection 95.183.13.232:52574 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41658] end connection 95.183.13.232:44212 (76 connections now open) 2016-11-23T10:44:58.185+0300 I NETWORK [conn41663] end connection 95.183.13.232:44254 (76 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn41606] end connection 95.183.13.232:43936 (75 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn62366] end connection 185.22.234.231:42318 (76 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn62359] end connection 185.22.234.231:42090 (76 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn32383] end connection 185.22.234.231:46922 (73 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn68315] end connection 95.183.13.232:58918 (76 connections now open) 2016-11-23T10:44:58.186+0300 I NETWORK [conn68393] end connection 95.183.13.232:59654 (75 connections now open) 2016-11-23T10:44:58.186+0300 I - [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 560 2016-11-23T10:44:58.189+0300 I - [ReplicationExecutor] ***aborting after invariant() failure 2016-11-23T10:44:58.214+0300 F - [ReplicationExecutor] Got signal: 6 (Aborted). 0x1322722 0x1321879 0x1322082 0x7fd6c1b82100 0x7fd6c17e65f7 0x7fd6c17e7ce8 0x12a86fb 0xede17a 0xf54a96 0xf0b249 0xf1f4b2 0xf24445 0x1b42e70 0x7fd6c1b7adc5 0x7fd6c18a7ced ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F22722","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F21879"},{"b":"400000","o":"F22082"},{"b":"7FD6C1B73000","o":"F100"},{"b":"7FD6C17B1000","o":"355F7","s":"gsignal"},{"b":"7FD6C17B1000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA86FB","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"ADE17A"},{"b":"400000","o":"B54A96","s":"_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE"},{"b":"400000","o":"B0B249","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb"},{"b":"400000","o":"B1F4B2"},{"b":"400000","o":"B24445","s":"_ZN5mongo4repl19ReplicationExecutor3runEv"},{"b":"400000","o":"1742E70","s":"execute_native_thread_routine"},{"b":"7FD6C1B73000","o":"7DC5"},{"b":"7FD6C17B1000","o":"F6CED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.36.1.el7.x86_64", "version" : "#1 SMP Sun Sep 18 13:04:29 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "6CC0DB94B5F0B88048BD857B35F6D91747E14577" }, { "b" : "7FFF5DCDC000", "elfType" : 3, "buildId" : "C30F6EAE7A15AD672A41176590C84B1BC35C0E02" }, { "b" : "7FD6C2A9B000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7FD6C26B3000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "42AAFD25E9B5F4CE2EFE6309491445B1A92A575D" }, { "b" : "7FD6C24AB000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "1D2AD4EAA62BAD560685A4B8DCCC8D9AA95E22CE" }, { "b" : "7FD6C22A7000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7FD6C1FA5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6DADD94D2A0885D50D09C465EA1970F23FB4629D" }, { "b" : "7FD6C1D8F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7FD6C1B73000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "DF6CCEE00C9F4C983A9464E43D17CC3311B51A8F" }, { "b" : "7FD6C17B1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "53C0918C85FA9CC08D2B57E76467631AB07554AE" }, { "b" : "7FD6C2D08000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F9AF7BA309F063D3BF9657A21436B4DCAC03CF07" }, { "b" : "7FD6C1565000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC2
- is related to
-
SERVER-28079 Secondary mongod crashes when removed from replicaset using rs.remove()
- Closed