|
I'm seeing something very similar. I'm using replication on MongoDB 3.2.5 and twice I've gotten Fatal Assertion 28546. I have a replica set with three databases.
2016-04-18T02:16:45.549+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 80, after asserts: 90, after connections: 100, after extra_info: 280, after globalLock: 290, after locks: 380, after network: 380, after opcounters: 380, after opcountersRepl: 380, after repl: 670, after storageEngine: 810, after tcmalloc: 900, after wiredTiger: 1230, at end: 1340 }
|
2016-04-18T02:16:46.537+0000 I COMMAND [conn423] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27015", fromId: 2, term: 37 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 396ms
|
2016-04-18T02:16:47.561+0000 I COMMAND [conn425] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27016", fromId: 1, term: 37 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 714ms
|
2016-04-18T02:16:48.285+0000 F REPL [rsBackgroundSync] sync producer exception: Resource temporarily unavailable
|
2016-04-18T02:16:48.475+0000 I - [rsBackgroundSync] Fatal Assertion 28546
|
2016-04-18T02:16:48.504+0000 I - [rsBackgroundSync]
|
|
***aborting after fassert() failure
|
|
|
2016-04-18T02:16:54.597+0000 I COMMAND [conn425] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27016", fromId: 1, term: 37 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 542ms
|
2016-04-18T02:16:54.601+0000 I REPL [ReplicationExecutor] Canceling priority takeover callback
|
2016-04-18T02:16:55.363+0000 I NETWORK [initandlisten] connection accepted from 127.0.0.1:48748 #430 (8 connections now open)
|
2016-04-18T02:16:55.928+0000 I REPL [ReplicationExecutor] Starting an election for a priority takeover
|
2016-04-18T02:16:56.312+0000 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected
|
2016-04-18T02:16:56.312+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 290, after asserts: 480, after connections: 810, after extra_info: 1830, after globalLock: 2310, after locks: 2450, after network: 2450, after opcounters: 2450, after opcountersRepl: 2450, after repl: 2820, after storageEngine: 3330, after tcmalloc: 4280, after wiredTiger: 5310, at end: 5773 }
|
2016-04-18T02:16:56.312+0000 F - [rsBackgroundSync] Got signal: 6 (Aborted).
|
|
0x12f3502 0x12f2659 0x12f2e62 0x7fb037374340 0x7fb036fd5cc9 0x7fb036fd90d8 0x127d9d2 0xe44646 0x7fb037b4fa40 0x7fb03736c182 0x7fb03709947d
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"EF3502","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EF2659"},{"b":"400000","o":"EF2E62"},{"b":"7FB037364000","o":"10340"},{"b":"7FB036F9F000","o":"36CC9","s":"gsignal"},{"b":"7FB036F9F000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E7D9D2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"A44646","s":"_ZN5mongo4repl14BackgroundSync14producerThreadEv"},{"b":"7FB037A9E000","o":"B1A40"},{"b":"7FB037364000","o":"8182"},{"b":"7FB036F9F000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.4", "gitVersion" : "e2ee9ffcf9f5a94fad76802e28cc978718bb7a30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-74-generic", "version" : "#118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EF46210F8976780D45B811C3540FECB9E734EABE" }, { "b" : "7FFFE7AAC000", "elfType" : 3, "buildId" : "DC075B751E9FB361F14CD59BD81300A6BB5CB377" }, { "b" : "7FB038589000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7FB0381AE000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7FB037FA6000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FB037DA2000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FB037A9E000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "4BF6F7ADD8244AD86008E6BF40D90F8873892197" }, { "b" : "7FB037798000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FB037582000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7FB037364000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FB036F9F000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FB0387E8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f3502]
|
mongod(+0xEF2659) [0x12f2659]
|
mongod(+0xEF2E62) [0x12f2e62]
|
libpthread.so.0(+0x10340) [0x7fb037374340]
|
libc.so.6(gsignal+0x39) [0x7fb036fd5cc9]
|
libc.so.6(abort+0x148) [0x7fb036fd90d8]
|
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127d9d2]
|
mongod(_ZN5mongo4repl14BackgroundSync14producerThreadEv+0x116) [0xe44646]
|
libstdc++.so.6(+0xB1A40) [0x7fb037b4fa40]
|
libpthread.so.0(+0x8182) [0x7fb03736c182]
|
libc.so.6(clone+0x6D) [0x7fb03709947d]
|
----- END BACKTRACE -----
|
2016-04-19T00:08:55.406+0000 I CONTROL [main] ***** SERVER RESTARTED *****
|
And a second time today 2016-05-02:
2016-05-02T06:29:43.430+0000 I COMMAND [conn1946] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27016", fromId: 1, term: 46 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:394 locks:{} protocol:op_command 771ms
|
2016-05-02T06:29:43.430+0000 I COMMAND [conn1947] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27015", fromId: 2, term: 46 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:394 locks:{} protocol:op_command 771ms
|
2016-05-02T06:29:43.835+0000 I REPL [ReplicationExecutor] syncing from: localhost:27016
|
2016-05-02T06:29:45.663+0000 I ACCESS [conn1958] Successfully authenticated as principal oplogger on admin
|
2016-05-02T06:29:46.262+0000 F REPL [rsBackgroundSync] sync producer exception: Resource temporarily unavailable
|
2016-05-02T06:29:46.361+0000 I REPL [ReplicationExecutor] Canceling priority takeover callback
|
2016-05-02T06:29:47.073+0000 I - [rsBackgroundSync] Fatal Assertion 28546
|
2016-05-02T06:29:47.075+0000 I COMMAND [conn1958] command admin.system.users command: saslContinue { saslContinue: 1, conversationId: 1, payload: BinData(0, ) } keyUpdates:0 writeConflicts:0 numYields:0 reslen:78 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 6704ms
|
2016-05-02T06:29:47.155+0000 I REPL [ReplicationExecutor] Starting an election for a priority takeover
|
2016-05-02T06:29:47.162+0000 I REPL [ReplicationExecutor] conducting a dry run election to see if we could be elected
|
2016-05-02T06:29:47.250+0000 I - [rsBackgroundSync]
|
|
***aborting after fassert() failure
|
|
|
2016-05-02T06:29:48.601+0000 I COMMAND [conn1946] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27016", fromId: 1, term: 46 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 281ms
|
2016-05-02T06:29:49.309+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 70, after asserts: 140, after connections: 140, after extra_info: 590, after globalLock: 790, after locks: 890, after network: 980, after opcounters: 980, after opcountersRepl: 980, after repl: 2980, after storageEngine: 2980, after tcmalloc: 3080, after wiredTiger: 3080, at end: 3620 }
|
2016-05-02T06:29:49.549+0000 I REPL [ReplicationExecutor] VoteRequester: Got no vote from localhost:27016 because: candidate's data is staler than mine, resp:{ term: 46, voteGranted: false, reason: "candidate's data is staler than mine", ok: 1.0 }
|
2016-05-02T06:29:51.012+0000 I COMMAND [conn1947] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27015", fromId: 2, term: 46 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 635ms
|
2016-05-02T06:29:52.360+0000 I COMMAND [conn1946] command admin.$cmd command: replSetHeartbeat { replSetHeartbeat: "mongodbreplica", configVersion: 2, from: "localhost:27016", fromId: 1, term: 46 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:425 locks:{} protocol:op_command 726ms
|
2016-05-02T06:29:58.242+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to localhost:27015
|
2016-05-02T06:29:58.244+0000 I REPL [ReplicationExecutor] Scheduling priority takeover at 2016-05-02T06:30:08.240+0000
|
2016-05-02T06:29:58.900+0000 F - [rsBackgroundSync] Got signal: 6 (Aborted).
|
|
0x1315022 0x1314179 0x1314982 0x7fc7e7ebe340 0x7fc7e7b1fcc9 0x7fc7e7b230d8 0x129f4f2 0xe64886 0x1b2b8d0 0x7fc7e7eb6182 0x7fc7e7be347d
|
----- BEGIN BACKTRACE -----
|
{"backtrace":[{"b":"400000","o":"F15022","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F14179"},{"b":"400000","o":"F14982"},{"b":"7FC7E7EAE000","o":"10340"},{"b":"7FC7E7AE9000","o":"36CC9","s":"gsignal"},{"b":"7FC7E7AE9000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E9F4F2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"A64886","s":"_ZN5mongo4repl14BackgroundSync14producerThreadEv"},{"b":"400000","o":"172B8D0","s":"execute_native_thread_routine"},{"b":"7FC7E7EAE000","o":"8182"},{"b":"7FC7E7AE9000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.5", "gitVersion" : "34e65e5383f7ea1726332cb175b73077ec4a1b02", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-74-generic", "version" : "#118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "8BD0E2ADD4592C91BBADCA1EEBC2B002DF5555A6" }, { "b" : "7FFC812EE000", "elfType" : 3, "buildId" : "DC075B751E9FB361F14CD59BD81300A6BB5CB377" }, { "b" : "7FC7E8DD0000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "E21720F2804EF30440F2B39CD409252C26F58F73" }, { "b" : "7FC7E89F4000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "9BC22F9457E3D7E9CF8DDC135C0DAC8F7742135D" }, { "b" : "7FC7E87EC000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7FC7E85E8000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7FC7E82E2000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7FC7E80CC000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7FC7E7EAE000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7FC7E7AE9000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7FC7E902F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}
|
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1315022]
|
mongod(+0xF14179) [0x1314179]
|
mongod(+0xF14982) [0x1314982]
|
libpthread.so.0(+0x10340) [0x7fc7e7ebe340]
|
libc.so.6(gsignal+0x39) [0x7fc7e7b1fcc9]
|
libc.so.6(abort+0x148) [0x7fc7e7b230d8]
|
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x129f4f2]
|
mongod(_ZN5mongo4repl14BackgroundSync14producerThreadEv+0x116) [0xe64886]
|
mongod(execute_native_thread_routine+0x20) [0x1b2b8d0]
|
libpthread.so.0(+0x8182) [0x7fc7e7eb6182]
|
libc.so.6(clone+0x6D) [0x7fc7e7be347d]
|
----- END BACKTRACE -----
|
2016-05-02T21:24:30.747+0000 I CONTROL [main] ***** SERVER RESTARTED *****
|
|