The global CommitJob object is destructed on ::exit() before thread execution is halted. This affects the v2.0 branch; master works around this by using a static reference.
To reproduce, send SIGQUIT to the mongod process while flooding a single connection with fire-and-forget writes:
> for (i=0;i<100000;i++) { db.test.insert({ a:1 }); }
The resulting crash varies. In the one below, the initAndListen thread receives the SIGQUIT. This thread then ::exit()s, starts destructing the mongo::dur::CommitJob static, generates a SIGSEGV when destructing a vector of Writes (from a TaskQueue<Writes>), then receives it's own SIGSEGV. The durability thread then fails an assertion on pthread_mutex_lock() while in mongo::dur::CommitJob::notifyCommitted()/NotifyAll(). Finally, an execCommand thread calls terminate() due to 'pure virtual method called', invokes myterminate, and gets a SIGABRT. At this point some threads continue to run and the process stops handling signals.
^\Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
Fri Apr 6 20:02:16 Got signal: 3 (Quit).
Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
...
Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
Fri Apr 6 20:02:16 [conn3] getmore local.oplog.rs query: { ts: { $gte: new Date(5728488334133562937) } } cursorid:8337457123675887220 nreturned:7 reslen:1091 0ms
Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
Fri Apr 6 20:02:16 Backtrace:
0x8a531c 0xc6b139 0x7f142694b420 0x7f14269f1613 0x8ca81b 0x94a844 0xc635e8 0xc64ef3 0xc65246 0xc69c25 0x7f142693630d 0x82d8c9
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x8a531c]
./mongod(_ZN5mongo10abruptQuitEi+0x110) [0xc6b139]
/lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f142694b420]
/lib/x86_64-linux-gnu/libc.so.6(__select+0x33) [0x7f14269f1613]
./mongod(_ZN5mongo8Listener13initAndListenEv+0x2f1) [0x8ca81b]
./mongod(_ZN5mongo17PortMessageServer3runEv+0x1c) [0x94a844]
./mongod(_ZN5mongo6listenEi+0x11d) [0xc635e8]
./mongod(_ZN5mongo14_initAndListenEi+0x650) [0xc64ef3]
./mongod(_ZN5mongo13initAndListenEi+0x18) [0xc65246]
./mongod(main+0x4668) [0xc69c25]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7f142693630d]
./mongod() [0x82d8c9]
Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
Logstream::get called in uninitialized state
Logstream::get called in uninitialized state
Fri Apr 6 20:02:16 [conn3] getmore local.oplog.rs query: { ts: { $gte: new Date(5728488334133562937) } } cursorid:8337457123675887220 nreturned:4 reslen:632 0ms
Fri Apr 6 20:02:16 [conn4] insert test.asdf 0ms
Logstream::get called in uninitialized state
Fri Apr 6 20:02:16 [conn4] Accessing: test for the first time
Fri Apr 6 20:02:16 [initandlisten] ERROR: Client::shutdown not called: initandlisten
Fri Apr 6 20:02:16 Invalid access at address: 0x9
Fri Apr 6 20:02:16 Got signal: 11 (Segmentation fault).
Fri Apr 6 20:02:16 Backtrace:
0x8a531c 0xc6b139 0xc6b314 0x7f1427d07060 0x7f142698f9e6 0x7f14269902a9 0x7f14269939cc 0xa78f1c 0xa78d40 0xa7966d 0xa79516 0xa792b1 0xa7933b 0xa79409 0x7f1426950821 0x7f14269508a5 0xc6b180 0x7f142694b420 0x7f14269f1613 0x8ca81b
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x8a531c]
./mongod(_ZN5mongo10abruptQuitEi+0x110) [0xc6b139]
./mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x103) [0xc6b314]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7f1427d07060]
/lib/x86_64-linux-gnu/libc.so.6(+0x7a9e6) [0x7f142698f9e6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7b2a9) [0x7f14269902a9]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c) [0x7f14269939cc]
./mongod(_ZN9__gnu_cxx13new_allocatorIN5mongo3dur6Writes1DEE10deallocateEPS4_m+0x20) [0xa78f1c]
./mongod(_ZNSt12_Vector_baseIN5mongo3dur6Writes1DESaIS3_EE13_M_deallocateEPS3_m+0x32) [0xa78d40]
./mongod(_ZNSt12_Vector_baseIN5mongo3dur6Writes1DESaIS3_EED1Ev+0x45) [0xa7966d]
./mongod(_ZNSt6vectorIN5mongo3dur6Writes1DESaIS3_EED1Ev+0x42) [0xa79516]
./mongod(_ZN5mongo9TaskQueueINS_3dur6Writes1DEED1Ev+0x4b) [0xa792b1]
./mongod(_ZN5mongo3dur6WritesD1Ev+0x4d) [0xa7933b]
./mongod(_ZN5mongo3dur9CommitJobD2Ev+0x2f) [0xa79409]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b821) [0x7f1426950821]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b8a5) [0x7f14269508a5]
./mongod(_ZN5mongo10abruptQuitEi+0x157) [0xc6b180]
/lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f142694b420]
/lib/x86_64-linux-gnu/libc.so.6(__select+0x33) [0x7f14269f1613]
./mongod(_ZN5mongo8Listener13initAndListenEv+0x2f1) [0x8ca81b]
mongod: /usr/include/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:26: boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*): Assertion `!pthread_mutex_lock(m)' failed.
Fri Apr 6 20:02:16 Got signal: 6 (Aborted).
Fri Apr 6 20:02:16 Backtrace:
0x8a531c 0xc6b139 0x7f142694b420 0x7f142694b3a5 0x7f142694eb0b 0x7f1426943d4d 0x850629 0x89e973 0x8c0202 0xa7775e 0xa75804 0xa7592c 0xa760b0 0xa763de 0xa7949f 0x7f14275dfba9 0x7f1427cfeefc 0x7f14269f859d
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x8a531c]
./mongod(_ZN5mongo10abruptQuitEi+0x110) [0xc6b139]
/lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f142694b420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f142694b3a5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f142694eb0b]
/lib/x86_64-linux-gnu/libc.so.6(__assert_fail+0xdd) [0x7f1426943d4d]
./mongod(_ZN5boost7pthread25pthread_mutex_scoped_lockC2EP15pthread_mutex_t+0x55) [0x850629]
./mongod(_ZN5boost22condition_variable_any10notify_allEv+0x1f) [0x89e973]
./mongod(_ZN5mongo9NotifyAll9notifyAllEy+0x4e) [0x8c0202]
./mongod(_ZN5mongo3dur9CommitJob15notifyCommittedEv+0x2a) [0xa7775e]
./mongod(_ZN5mongo3dur28_groupCommitWithLimitedLocksEv+0x1ae) [0xa75804]
./mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x10) [0xa7592c]
./mongod() [0xa760b0]
./mongod(_ZN5mongo3dur9durThreadEv+0x156) [0xa763de]
./mongod(_ZN5boost6detail11thread_dataIPFvvEE3runEv+0x19) [0xa7949f]
/usr/lib/libboost_thread.so.1.46.1(thread_proxy+0x69) [0x7f14275dfba9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f1427cfeefc]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f14269f859d]
Logstream::get called in uninitialized state
Fri Apr 6 20:02:17 [conn2] runQuery called admin.$cmd { replSetHeartbeat: "rs1", v: 1, pv: 1, checkEmpty: false, from: "localhost:27018" }
Logstream::get called in uninitialized state
Fri Apr 6 20:02:17 [conn2] run command admin.$cmd { replSetHeartbeat: "rs1", v: 1, pv: 1, checkEmpty: false, from: "localhost:27018" }
pure virtual method called
Fri Apr 6 20:02:17 terminate() called, printing stack:
0x8a531c 0xc6b496 0x7f1427aa9f26 0x7f1427aa9f53 0x7f1427aaa96f 0xbc87e0 0xbc91d4 0xbaa73a 0xbabb0a 0xb34962 0xb35364 0xc6c0c5 0x949a0e 0x7f1427cfeefc 0x7f14269f859d
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x8a531c]
./mongod(_ZN5mongo11myterminateEv+0x5f) [0xc6b496]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9f26) [0x7f1427aa9f26]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9f53) [0x7f1427aa9f53]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xba96f) [0x7f1427aaa96f]
./mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x630) [0xbc87e0]
./mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x367) [0xbc91d4]
./mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x46) [0xbaa73a]
./mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x24e) [0xbabb0a]
./mongod() [0xb34962]
./mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x335) [0xb35364]
./mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xe1) [0xc6c0c5]
./mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x2b2) [0x949a0e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f1427cfeefc]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f14269f859d]
Fri Apr 6 20:02:17 Got signal: 6 (Aborted).
Fri Apr 6 20:02:17 Backtrace:
0x8a531c 0xc6b139 0x7f142694b420 0x7f142694b3a5 0x7f142694eb0b 0xc6b49b 0x7f1427aa9f26 0x7f1427aa9f53 0x7f1427aaa96f 0xbc87e0 0xbc91d4 0xbaa73a 0xbabb0a 0xb34962 0xb35364 0xc6c0c5 0x949a0e 0x7f1427cfeefc 0x7f14269f859d
./mongod(_ZN5mongo15printStackTraceERSo+0x27) [0x8a531c]
./mongod(_ZN5mongo10abruptQuitEi+0x110) [0xc6b139]
/lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f142694b420]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f142694b3a5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f142694eb0b]
./mongod(_ZN5mongo11myterminateEv+0x64) [0xc6b49b]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9f26) [0x7f1427aa9f26]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb9f53) [0x7f1427aa9f53]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xba96f) [0x7f1427aaa96f]
./mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x630) [0xbc87e0]
./mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x367) [0xbc91d4]
./mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x46) [0xbaa73a]
./mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x24e) [0xbabb0a]
./mongod() [0xb34962]
./mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x335) [0xb35364]
./mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xe1) [0xc6c0c5]
./mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x2b2) [0x949a0e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f1427cfeefc]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f14269f859d]
Logstream::get called in uninitialized state
Fri Apr 6 20:02:55 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms
...
Fri Apr 6 20:12:55 [PeriodicTask::Runner] task: DBConnectionPool-cleaner took: 0ms
Logstream::get called in uninitialized state
Fri Apr 6 20:13:03 [conn4] couldn't get lazy rwlock
Logstream::get called in uninitialized state
- is related to
-
SERVER-5566 abruptQuit should use _exit
-
- Closed
-