-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.2.11, 3.4.2, 3.5.3
-
Component/s: Replication
-
Replication
-
ALL
-
v3.6, v3.4, v3.2
-
-
(copied to CRM)
Hi,
When I try to remove a replica from the replicaset using rs.remove() command, the secondary when we removed gets crashed.
Say, I run rs.remove(10.0.1.211) on the primary node.
The mongod running at 10.0.1.211 get crashed with the below trace:
2017-02-23T02:46:40.516-0800 I - [ReplicationExecutor] Invariant failure i < _members.size() src/mongo/db/repl/replica_set_config.cpp 560 2017-02-23T02:46:40.516-0800 I - [ReplicationExecutor] ***aborting after invariant() failure 2017-02-23T02:46:40.522-0800 F - [ReplicationExecutor] Got signal: 6 (Aborted). 0x132fa32 0x132eb89 0x132f392 0x7f001f5375b0 0x7f001f1b6f49 0x7f001f1b8348 0x12b581b 0xee879a 0xf60936 0xf16939 0xf2ae32 0xf2fdc5 0x1b5f300 0x7f001f52ff18 0x7f001f265e9d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"F2FA32","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F2EB89"},{"b":"400000","o":"F2F392"},{"b":"7F001F528000","o":"F5B0"},{"b":"7F001F183000","o":"33F49","s":"gsignal"},{"b":"7F001F183000","o":"35348","s":"abort"},{"b":"400000","o":"EB581B","s":"_ZN5mongo15invariantFailedEPKcS1_j"},{"b":"400000","o":"AE879A"},{"b":"400000","o":"B60936","s":"_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE"},{"b":"400000","o":"B16939","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb"},{"b":"400000","o":"B2AE32"},{"b":"400000","o":"B2FDC5","s":"_ZN5mongo4repl19ReplicationExecutor3runEv"},{"b":"400000","o":"175F300","s":"execute_native_thread_routine"},{"b":"7F001F528000","o":"7F18"},{"b":"7F001F183000","o":"E2E9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.23-31.54.amzn1.x86_64", "version" : "#1 SMP Tue Oct 18 22:02:09 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "88BEAACF5E9DF7E467AF167F9F4F31D20015315F" }, { "b" : "7FFC6E5F6000", "elfType" : 3, "buildId" : "3C621354FA6866C1B7DBFFDA88CE59560C001BE1" }, { "b" : "7F0020447000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "FA7CA2477D0B7E4D5D3D875501F1EEC5C2D883A2" }, { "b" : "7F0020064000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "AEC6000432F3A98B6F7F3BD3793F12B13D0A0FC2" }, { "b" : "7F001FE5C000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "92B6FB6A7CF87B575FE6043F95639C1A081E2E2A" }, { "b" : "7F001FC58000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "F08BBD07F4042BC8D0D314F5E1F7F5D26F028CFB" }, { "b" : "7F001F95A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "DFB15C9F2E7C575E1954C19CEFC2842DE2C265DB" }, { "b" : "7F001F744000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DB655E06F0F4F7B4EC561BB7E620F5D5BC4F1C54" }, { "b" : "7F001F528000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3C51D8CB39ED16242013CB77B0125707C6F34406" }, { "b" : "7F001F183000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3B97A7F435805785DFD11096836F4E904FFF2599" }, { "b" : "7F00206B3000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "CAD953A4B324B3E3AA1449742558362857E826F2" }, { "b" : "7F001EF40000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9DF61878D8918F25CC74AD01F417FDB051DFE3DA" }, { "b" : "7F001EC5B000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "6F1DB0F811D1B210520443442D4437BC43BF9A80" }, { "b" : "7F001EA58000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "E52249AE6C9865B5C3B9697A57FC92200DA51CF3" }, { "b" : "7F001E82D000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "F7DF34078FD7BFD684FE46D5F677EEDA1D9B9DC9" }, { "b" : "7F001E617000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "87B4EBF2183C8EA4AB657212203EFFE6340E2F4F" }, { "b" : "7F001E40C000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "381960ACAB9C39461D58BDE7B272C4F61BB3582F" }, { "b" : "7F001E209000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "BF48CD5658DE95CE058C4B828E81C97E2AE19643" }, { "b" : "7F001DFF2000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "E25410C0CBAA4D33369EDD8086A0CF24F5AFE4E7" }, { "b" : "7F001DDD1000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "803D7EF21A989677D056E52BAEB9AB5B154FB9D9" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x132fa32] mongod(+0xF2EB89) [0x132eb89] mongod(+0xF2F392) [0x132f392] libpthread.so.0(+0xF5B0) [0x7f001f5375b0] libc.so.6(gsignal+0x39) [0x7f001f1b6f49] libc.so.6(abort+0x148) [0x7f001f1b8348] mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0x12b581b] mongod(+0xAE879A) [0xee879a] mongod(_ZNK5mongo4repl23TopologyCoordinatorImpl22shouldChangeSyncSourceERKNS_11HostAndPortERKNS0_6OpTimeES7_bNS_6Date_tE+0x1B6) [0xf60936] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl23_shouldChangeSyncSourceERKNS_8executor12TaskExecutor12CallbackArgsERKNS_11HostAndPortERKNS0_6OpTimeEbPb+0x89) [0xf16939] mongod(+0xB2AE32) [0xf2ae32] mongod(_ZN5mongo4repl19ReplicationExecutor3runEv+0x275) [0xf2fdc5] mongod(execute_native_thread_routine+0x20) [0x1b5f300] libpthread.so.0(+0x7F18) [0x7f001f52ff18] libc.so.6(clone+0x6D) [0x7f001f265e9d] ----- END BACKTRACE -----
- is related to
-
SERVER-30089 Arbiter crash with invariant failure i < _members.size()
- Closed
- related to
-
SERVER-27166 secondary crashes after being removed from the replica set
- Closed