Show
Set ulimit -u to 1k. Added some requests to mongodb, opening new connections (near 1k)
For some time mongod was working as expected, it was throwing errors like:
2017-09-30T11:13:48.152+0300 I NETWORK [thread1] connection accepted from [XXX]:47920 #1881 (930 connections now open)
2017-09-30T11:13:48.152+0300 I - [thread1] pthread_create failed: Resource temporarily unavailable
2017-09-30T11:13:48.152+0300 I - [thread1] failed to create service entry worker thread for [XXX]:47920
But suddenly it throws such error:
2017-09-30T11:13:48.191+0300 I REPL [rsBackgroundSync] sync source candidate: YYYY
2017-09-30T11:13:48.191+0300 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to YYYY
2017-09-30T11:13:48.191+0300 F ASIO [NetworkInterfaceASIO-RS-0] Uncaught exception in NetworkInterfaceASIO IO worker thread of type: UnknownError: Caught std::exception of type std::system_error: thread: Resource temporarily unavailable
2017-09-30T11:13:48.191+0300 I - [NetworkInterfaceASIO-RS-0] Fatal Assertion 28820 at src/mongo/executor/network_interface_asio.cpp 168
2017-09-30T11:13:48.191+0300 I - [NetworkInterfaceASIO-RS-0]
***aborting after fassert() failure
2017-09-30T11:13:48.195+0300 F - [NetworkInterfaceASIO-RS-0] Got signal: 6 (Aborted).
0x56467a204f31 0x56467a204029 0x56467a20450d 0x7faa6adc1cb0 0x7faa6aa27035 0x7faa6aa2a79b 0x5646794c7b59 0x564679f80f3a 0x56467ac73790 0x7faa6adb9e9a 0x7faa6aae72ed
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"564678CCC000","o":"1538F31","s":"_ZN5mongo15printStackTraceERSo"},{"b":"564678CCC000","o":"1538029"},{"b":"564678CCC000","o":"153850D"},{"b":"7FAA6ADB2000","o":"FCB0"},{"b":"7FAA6A9F1000","o":"36035","s":"gsignal"},{"b":"7FAA6A9F1000","o":"3979B","s":"abort"},{"b":"564678CCC000","o":"7FBB59","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"564678CCC000","o":"12B4F3A"},{"b":"564678CCC000","o":"1FA7790"},{"b":"7FAA6ADB2000","o":"7E9A"},{"b":"7FAA6A9F1000","o":"F62ED","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.3", "gitVersion" : "f07437fb5a6cca07c10bafa78365456eb1d6d5e1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.68-33", "version" : "#1 SMP Mon May 15 15:50:23 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "564678CCC000", "elfType" : 3, "buildId" : "7F4B6253104ED1F56A3EBE57764D712224D2EE74" }, { "b" : "7FFF0C18E000", "elfType" : 3, "buildId" : "EAE1311A713DED3C92BFD94265DF468A7189F7CD" }, { "b" : "7FAA6B6E5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "99255CAB5455CB9EDFA0270CDAA6B6A7BBEF2E1B" }, { "b" : "7FAA6B4E1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "A51A5921F4E05E4D20B165D398BA4D563960DA9A" }, { "b" : "7FAA6B1E5000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "6D3D633C88F7E9835D180ACE648CEDB21C8021B7" }, { "b" : "7FAA6AFCF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7FAA6ADB2000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9B1F69F5DC3A6820BB3CA4B2DB147ABAA486A41A" }, { "b" : "7FAA6A9F1000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "68FC0E76A868E47807E3604B02D8BAA580A4E2CB" }, { "b" : "7FAA6B8ED000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2B91CD40CE35626DAB827FEEE08F671253FA7B88" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x56467a204f31]
mongod(+0x1538029) [0x56467a204029]
mongod(+0x153850D) [0x56467a20450d]
libpthread.so.0(+0xFCB0) [0x7faa6adc1cb0]
libc.so.6(gsignal+0x35) [0x7faa6aa27035]
libc.so.6(abort+0x17B) [0x7faa6aa2a79b]
mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5646794c7b59]
mongod(+0x12B4F3A) [0x564679f80f3a]
mongod(+0x1FA7790) [0x56467ac73790]
libpthread.so.0(+0x7E9A) [0x7faa6adb9e9a]
libc.so.6(clone+0x6D) [0x7faa6aae72ed]
----- END BACKTRACE -----
This was happening with every replica in replicaset (because of constant number of requests), replicaset became unstable.
Bug
Major - P3