Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27166

secondary crashes after being removed from the replica set

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Admin
    • Labels:
      None
    • ALL

      the ex-secondary crashed about the same time it was removed from the RS configuration.
      I'm sure that it is not an intended behaviour (formerly it just entered "OTHER" state).

      centos7, mongo 3.2.9

      Extract from the mongod.log

      2016-11-23T10:44:58.160+0300 I REPL     [SyncSourceFeedback] SyncSourceFeedback error sending update, response: { configVersion: 156007, ok: 0.0, errmsg: "Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007", code: 93 }
      2016-11-23T10:44:58.160+0300 I REPL     [SyncSourceFeedback] updateUpstream failed: InvalidReplicaSetConfig: Received replSetUpdatePosition for node with memberId 0 whose config version of 156006 doesn't match our config version of 156007, will retry
      2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] Cannot find self in new replica set configuration; I must be removed; NodeNotFound: No host described in new configuration 156007 for replica set rs3 maps to this node
      2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] New replica set config in use: { _id: "rs3", version: 156007, protocolVersion: 1, members: [ { _id: 1, host: "uaDbReplica:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 }, { _id: 2, host: "uaDb1:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('582643d6ef9bf5817d75fea5') } }
      2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] This node is not a member of the config
      2016-11-23T10:44:58.182+0300 I REPL     [ReplicationExecutor] transition to REMOVED
      2016-11-23T10:44:58.183+0300 I NETWORK  [conn41592] end connection 95.183.13.232:43834 (76 connections now open)
      2016-11-23T10:44:58.183+0300 I NETWORK  [conn41591] end connection 95.183.13.232:43816 (76 connections now open)
      2016-11-23T10:44:58.183+0300 I NETWORK  [conn61491] end connection 185.22.234.231:46820 (76 connections now open)
      2016-11-23T10:44:58.183+0300 I NETWORK  [conn61483] end connection 185.22.234.231:45884 (76 connections now open)
      2016-11-23T10:44:58.183+0300 I NETWORK  [conn41607] end connection 95.183.13.232:43948 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61485] end connection 185.22.234.231:46214 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61633] end connection 185.22.234.231:48156 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41574] end connection 37.143.15.188:51401 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn62243] end connection 185.22.234.231:51252 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61497] end connection 185.22.234.231:48028 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41622] end connection 95.183.13.232:44058 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41610] end connection 95.183.13.232:43980 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61486] end connection 185.22.234.231:46220 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61363] end connection 185.22.234.231:39466 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41624] end connection 95.183.13.232:44082 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61358] end connection 185.22.234.231:39170 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41648] end connection 95.183.13.232:44176 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61490] end connection 185.22.234.231:46804 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41677] end connection 95.183.13.232:44360 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41667] end connection 95.183.13.232:44296 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn62174] end connection 185.22.234.231:47910 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41594] end connection 95.183.13.232:43856 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn68128] end connection 95.183.13.232:58190 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62176] end connection 185.22.234.231:48068 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61487] end connection 185.22.234.231:46562 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61360] end connection 185.22.234.231:39428 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61492] end connection 185.22.234.231:46838 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41670] end connection 95.183.13.232:44316 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41657] end connection 95.183.13.232:44202 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41662] end connection 95.183.13.232:44244 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41673] end connection 95.183.13.232:44336 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn68576] end connection 95.183.13.232:60564 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41665] end connection 95.183.13.232:44276 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41639] end connection 95.183.13.232:44120 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41672] end connection 95.183.13.232:44334 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62364] end connection 185.22.234.231:42286 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn62246] end connection 185.22.234.231:52566 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41644] end connection 95.183.13.232:44152 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61495] end connection 185.22.234.231:47484 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62365] end connection 185.22.234.231:42314 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62358] end connection 185.22.234.231:41756 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41661] end connection 95.183.13.232:44232 (76 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn41581] end connection 95.183.13.232:43756 (73 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41664] end connection 95.183.13.232:44266 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41647] end connection 95.183.13.232:44164 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61359] end connection 185.22.234.231:39200 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62175] end connection 185.22.234.231:48010 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61496] end connection 185.22.234.231:47774 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41626] end connection 95.183.13.232:44106 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41593] end connection 95.183.13.232:43844 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61488] end connection 185.22.234.231:46564 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41690] end connection 95.183.13.232:44374 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61482] end connection 185.22.234.231:45824 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn61494] end connection 185.22.234.231:47188 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41623] end connection 95.183.13.232:44070 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41609] end connection 95.183.13.232:43970 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41669] end connection 95.183.13.232:44306 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61361] end connection 185.22.234.231:39434 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61489] end connection 185.22.234.231:46566 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn41655] end connection 95.183.13.232:44190 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61362] end connection 185.22.234.231:39464 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61632] end connection 185.22.234.231:48020 (76 connections now open)
      2016-11-23T10:44:58.184+0300 I NETWORK  [conn61631] end connection 185.22.234.231:47786 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62362] end connection 185.22.234.231:42278 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn68596] end connection 95.183.13.232:60810 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41642] end connection 95.183.13.232:44142 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41666] end connection 95.183.13.232:44286 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn62361] end connection 185.22.234.231:42250 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn64811] end connection 95.183.13.232:52574 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41658] end connection 95.183.13.232:44212 (76 connections now open)
      2016-11-23T10:44:58.185+0300 I NETWORK  [conn41663] end connection 95.183.13.232:44254 (76 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn41606] end connection 95.183.13.232:43936 (75 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn62366] end connection 185.22.234.231:42318 (76 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn62359] end connection 185.22.234.231:42090 (76 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn32383] end connection 185.22.234.231:46922 (73 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn68315] end connection 95.183.13.232:58918 (76 connections now open)
      2016-11-23T10:44:58.186+0300 I NETWORK  [conn68393] end connection 95.183.13.232:59654 (75 connections now open)
      2016-11-23T10:44:58.186+0300 I -        [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 560
      2016-11-23T10:44:58.189+0300 I -        [ReplicationExecutor]
      
      ***aborting after invariant() failure
      
      
      2016-11-23T10:44:58.214+0300 F -        [ReplicationExecutor] Got signal: 6 (Aborted).
      
       0x1322722 0x1321879 0x1322082 0x7fd6c1b82100 0x7fd6c17e65f7 0x7fd6c17e7ce8 0x12a86fb 0xede17a 0xf54a96 0xf0b249 0xf1f4b2 0xf24445 0x1b42e70 0x7fd6c1b7adc5 0x7fd6c18a7ced
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F22722","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F21879"},{"b":"400000","o":"F22082"},{"b":"7FD6C1B73000","o":"F100"},{"b":"7FD6C17B1000","o":"355F7","s":"gsignal"},{"b":"7FD6C17B1000","o":"36CE8","s":"abort"},{"b":"400000","o":"EA86FB","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"ADE17A"},{"b":"400000","o":"B54A96","s":"_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE"},{"b":"400000","o":"B0B249","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb"},{"b":"400000","o":"B1F4B2"},{"b":"400000","o":"B24445","s":"_ZN5mongo4repl19ReplicationExecutor3runEv"},{"b":"400000","o":"1742E70","s":"execute_native_thread_routine"},{"b":"7FD6C1B73000","o":"7DC5"},{"b":"7FD6C17B1000","o":"F6CED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.36.1.el7.x86_64", "version" : "#1 SMP Sun Sep 18 13:04:29 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "6CC0DB94B5F0B88048BD857B35F6D91747E14577" }, { "b" : "7FFF5DCDC000", "elfType" : 3, "buildId" : "C30F6EAE7A15AD672A41176590C84B1BC35C0E02" }, { "b" : "7FD6C2A9B000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "478D01A08B923A251D755BB421F3EBAF9F2982C1" }, { "b" : "7FD6C26B3000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "42AAFD25E9B5F4CE2EFE6309491445B1A92A575D" }, { "b" : "7FD6C24AB000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "1D2AD4EAA62BAD560685A4B8DCCC8D9AA95E22CE" }, { "b" : "7FD6C22A7000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "091060A163E7EDA25572F3B1BAF2E8F80209C00E" }, { "b" : "7FD6C1FA5000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6DADD94D2A0885D50D09C465EA1970F23FB4629D" }, { "b" : "7FD6C1D8F000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6AA1DCC4DE7F1836344949857FC2017278631FFD" }, { "b" : "7FD6C1B73000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "DF6CCEE00C9F4C983A9464E43D17CC3311B51A8F" }, { "b" : "7FD6C17B1000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "53C0918C85FA9CC08D2B57E76467631AB07554AE" }, { "b" : "7FD6C2D08000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F9AF7BA309F063D3BF9657A21436B4DCAC03CF07" }, { "b" : "7FD6C1565000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "D46A230FFF4A7B808B3CFC2
      

            Assignee:
            Unassigned Unassigned
            Reporter:
            hq9000 Sergey Grechin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: