Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-25247

MongoDB primary node crashes after SIGINT (ctrl+c)

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 3.2.6, 3.2.7, 3.2.8
    • Component/s: WiredTiger
    • Labels:
    • Environment:
      Ubuntu 15.10, AMD64
    • ALL
    • 0

      After sending SIGINT to the primary node of the replica set (3 node dev setup: primary, secondary, arbiter) mongod crashed with SIGSEGV in __wt_split_stash_discard_all().

      Log from primary at the time of crash is in the attachment

      SegvAnalysis from Apport:

      Segfault happened at: 0x19e0bf0 <__wt_split_stash_discard_all+48>:	cmpq   $0x0,(%rbx)
PC (0x019e0bf0) ok
source "$0x0" ok
destination "(%rbx)" (0x7fba414a9357) not located in a known VMA region (needed writable region)!
Stack memory exhausted (SP below stack segment)

      Stacktrace:

      #0  0x00000000019e0bf0 in __wt_split_stash_discard_all ()
No symbol table info available.
#1  0x0000000001a078af in __wt_connection_close ()
No symbol table info available.
#2  0x00000000019fd1e0 in __conn_close ()
No symbol table info available.
#3  0x0000000001085db3 in mongo::WiredTigerKVEngine::cleanShutdown() ()
No symbol table info available.
#4  0x0000000000facdd8 in mongo::ServiceContextMongoD::shutdownGlobalStorageEngineCleanly() ()
No symbol table info available.
#5  0x0000000000cd1210 in mongo::exitCleanly(mongo::ExitCode) ()
No symbol table info available.
#6  0x000000000131b8c1 in mongo::(anonymous namespace)::signalProcessingThread() ()
No symbol table info available.
#7  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#8  0x00007fc9564e06aa in start_thread (arg=0x7fc9558c2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc9558c2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502700402432, 5874792978760040640, 0, 140734797417775, 140502700403136, 140734797418552, -5902567582192140096, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#9  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

      Thread stacktrace:

      .
Thread 15 (Thread 0x7fc94c0af700 (LWP 2551)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
No symbol table info available.
#2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
No symbol table info available.
#3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
No symbol table info available.
#4  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#5  0x00007fc9564e06aa in start_thread (arg=0x7fc94c0af700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94c0af700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502540941056, 5874792978760040640, 0, 140502582899919, 140502540941760, 34713112, -5902551138372976448, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 14 (Thread 0x7fc94b0ad700 (LWP 2553)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
No symbol table info available.
#2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
No symbol table info available.
#3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
No symbol table info available.
#4  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#5  0x00007fc9564e06aa in start_thread (arg=0x7fc94b0ad700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94b0ad700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502524155648, 5874792978760040640, 0, 140502582899919, 140502524156352, 34713112, -5902562134562996032, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 13 (Thread 0x7fc94f8b6700 (LWP 2544)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
No symbol table info available.
#2  0x0000000001273c48 in mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>::deadlineMonitorThread() ()
No symbol table info available.
#3  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#4  0x00007fc9564e06aa in start_thread (arg=0x7fc94f8b6700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94f8b6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502599689984, 5874792978760040640, 0, 140734797414975, 140502599690688, 140734797417680, -5902554450329632576, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 12 (Thread 0x7fc9518ba700 (LWP 2540)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x0000000000bb77f9 in mongo::CondVarLockGrantNotification::wait(unsigned int) ()
No symbol table info available.
#2  0x0000000000bbb2c7 in mongo::LockerImpl<false>::lockComplete(mongo::ResourceId, mongo::LockMode, unsigned int, bool) ()
No symbol table info available.
#3  0x0000000000bb7e9c in mongo::LockerImpl<false>::lockGlobal(mongo::LockMode, unsigned int) ()
No symbol table info available.
#4  0x0000000000bb7b6d in mongo::LockerImpl<false>::restoreLockState(mongo::Locker::LockSnapshot const&) ()
No symbol table info available.
#5  0x0000000001092240 in mongo::WiredTigerRecordStore::yieldAndAwaitOplogDeletionRequest(mongo::OperationContext*) ()
No symbol table info available.
#6  0x00000000010981a3 in mongo::(anonymous namespace)::WiredTigerRecordStoreThread::_deleteExcessDocuments() ()
No symbol table info available.
#7  0x0000000001098718 in mongo::(anonymous namespace)::WiredTigerRecordStoreThread::run() ()
No symbol table info available.
#8  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
No symbol table info available.
#9  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#10 0x00007fc9564e06aa in start_thread (arg=0x7fc9518ba700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc9518ba700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502633260800, 5874792978760040640, 0, 140734797411743, 140502633261504, 140734797412240, -5902576434119737152, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#11 0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 11 (Thread 0x7fc94d8b2700 (LWP 2548)):
#0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
        ts = {tv_sec = 8, tv_nsec = 470999445}
        set = {__val = {65536, 0 <repeats 15 times>}}
        oset = {__val = {8405507, 140502566115200, 140502566115024, 67575832, 140827780, 140827776, 0, 0, 0, 69083232, 67951712, 140501265154048, 129257520, 129258768, 129257520, 2}}
        result = <optimized out>
#2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
No symbol table info available.
#3  0x0000000001326aee in mongo::sleepsecs(int) ()
No symbol table info available.
#4  0x00000000010a917b in mongo::TTLMonitor::run() ()
No symbol table info available.
#5  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
No symbol table info available.
#6  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#7  0x00007fc9564e06aa in start_thread (arg=0x7fc94d8b2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94d8b2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502566119168, 5874792978760040640, 0, 140734797414959, 140502566119872, 140734797417680, -5902550050135637824, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#8  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 10 (Thread 0x7fc94c8b0700 (LWP 2550)):
#0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
        ts = {tv_sec = 2, tv_nsec = 813136127}
        set = {__val = {65536, 0 <repeats 15 times>}}
        oset = {__val = {8405507, 65122808, 140502549329567, 140502549329632, 140502549329600, 10190813, 140502549329632, 14, 65122560, 0, 140502549329664, 140502549329904, 140502549329856, 11499820, 67783952, 140502549329632}}
        result = <optimized out>
#2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
No symbol table info available.
#3  0x0000000001326aee in mongo::sleepsecs(int) ()
No symbol table info available.
#4  0x0000000000b14fd8 in mongo::ClientCursorMonitor::run() ()
No symbol table info available.
#5  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
No symbol table info available.
#6  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#7  0x00007fc9564e06aa in start_thread (arg=0x7fc94c8b0700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94c8b0700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502549333760, 5874792978760040640, 0, 140734797414959, 140502549334464, 140734797417680, -5902552237347733312, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#8  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 9 (Thread 0x7fc94b8ae700 (LWP 2552)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x00000000012abb9e in mongo::(anonymous namespace)::PeriodicTaskRunner::run() ()
No symbol table info available.
#2  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
No symbol table info available.
#3  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#4  0x00007fc9564e06aa in start_thread (arg=0x7fc94b8ae700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94b8ae700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502532548352, 5874792978760040640, 0, 140734797414911, 140502532549056, 140734797417680, -5902563233537752896, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 8 (Thread 0x7fc94a8ac700 (LWP 2554)):
#0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
        ts = {tv_sec = 8, tv_nsec = 147663717}
        set = {__val = {65536, 0 <repeats 15 times>}}
        oset = {__val = {8405507, 0 <repeats 15 times>}}
        result = <optimized out>
#2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
No symbol table info available.
#3  0x0000000001326aee in mongo::sleepsecs(int) ()
No symbol table info available.
#4  0x00000000012bf49e in mongo::HostnameCanonicalizationWorker::_doWork() ()
No symbol table info available.
#5  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#6  0x00007fc9564e06aa in start_thread (arg=0x7fc94a8ac700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94a8ac700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502515762944, 5874792978760040640, 0, 140734797414943, 140502515763648, 140734797417680, -5902565429339782976, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 7 (Thread 0x7fc94f0b5700 (LWP 2545)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x0000000000e56b43 in mongo::RangeDeleter::doWork() ()
No symbol table info available.
#2  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#3  0x00007fc9564e06aa in start_thread (arg=0x7fc94f0b5700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94f0b5700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502591297280, 5874792978760040640, 0, 140734797415039, 140502591297984, 140734797417680, -5902553351354875712, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 6 (Thread 0x7fc94d0b1700 (LWP 2549)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
No symbol table info available.
#2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
No symbol table info available.
#3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
No symbol table info available.
#4  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#5  0x00007fc9564e06aa in start_thread (arg=0x7fc94d0b1700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc94d0b1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502557726464, 5874792978760040640, 0, 140502582899919, 140502557727168, 34713112, -5902548951160880960, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 5 (Thread 0x7fc946e99700 (LWP 2564)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001338af0 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) ()
No symbol table info available.
#2  0x0000000001338da1 in asio::detail::scheduler::run(std::error_code&) ()
No symbol table info available.
#3  0x000000000133cf3f in asio::io_service::run() ()
No symbol table info available.
#4  0x00000000013306e0 in asio_detail_posix_thread_function ()
No symbol table info available.
#5  0x00007fc9564e06aa in start_thread (arg=0x7fc946e99700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc946e99700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502454867712, 5874792978760040640, 0, 140502574506895, 140502454868416, 140502574507232, -5902538209447673664, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 4 (Thread 0x7fc9578d0cc0 (LWP 2530)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fc9564e2cfd in __GI___pthread_mutex_lock (mutex=0x20b4340 <mongo::shutdownLock>) at ../nptl/pthread_mutex_lock.c:80
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 0
        id = <optimized out>
#2  0x0000000000cd0eb5 in mongo::exitCleanly(mongo::ExitCode) ()
No symbol table info available.
#3  0x000000000096e054 in main ()
No symbol table info available.
.
Thread 3 (Thread 0x7fc934270700 (LWP 24993)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
No locals.
#1  0x0000000000ae5e77 in mongo::CappedInsertNotifier::_wait(std::unique_lock<std::mutex>&, unsigned long, std::chrono::duration<long, std::ratio<1l, 1000000l> >) const ()
No symbol table info available.
#2  0x0000000000ae8699 in mongo::CappedInsertNotifier::wait(unsigned long, std::chrono::duration<long, std::ratio<1l, 1000000l> >) const ()
No symbol table info available.
#3  0x0000000000b5389f in mongo::GetMoreCmd::run(mongo::OperationContext*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&) ()
No symbol table info available.
#4  0x0000000000bc7f93 in mongo::Command::run(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
No symbol table info available.
#5  0x0000000000bc8e24 in mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
No symbol table info available.
#6  0x0000000000b24ce0 in mongo::runCommands(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
No symbol table info available.
#7  0x0000000000cd6a15 in mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&) ()
No symbol table info available.
#8  0x00000000009b937c in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) ()
No symbol table info available.
#9  0x00000000012c9645 in mongo::PortMessageServer::handleIncomingMsg(void*) ()
No symbol table info available.
#10 0x00007fc9564e06aa in start_thread (arg=0x7fc934270700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc934270700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502140126976, 5874792978760040640, 0, 140734797414223, 140502140127680, 8388608, -5902779458297560896, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#11 0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 2 (Thread 0x7fc935575700 (LWP 2672)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000000001338af0 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) ()
No symbol table info available.
#2  0x0000000001338da1 in asio::detail::scheduler::run(std::error_code&) ()
No symbol table info available.
#3  0x000000000133cf3f in asio::io_service::run() ()
No symbol table info available.
#4  0x00000000013306e0 in asio_detail_posix_thread_function ()
No symbol table info available.
#5  0x00007fc9564e06aa in start_thread (arg=0x7fc935575700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc935575700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502160070400, 5874792978760040640, 0, 140502471648143, 8388608, 124960064, -5902776861452959552, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.
.
Thread 1 (Thread 0x7fc9558c2700 (LWP 2531)):
#0  0x00000000019e0bf0 in __wt_split_stash_discard_all ()
No symbol table info available.
#1  0x0000000001a078af in __wt_connection_close ()
No symbol table info available.
#2  0x00000000019fd1e0 in __conn_close ()
No symbol table info available.
#3  0x0000000001085db3 in mongo::WiredTigerKVEngine::cleanShutdown() ()
No symbol table info available.
#4  0x0000000000facdd8 in mongo::ServiceContextMongoD::shutdownGlobalStorageEngineCleanly() ()
No symbol table info available.
#5  0x0000000000cd1210 in mongo::exitCleanly(mongo::ExitCode) ()
No symbol table info available.
#6  0x000000000131b8c1 in mongo::(anonymous namespace)::signalProcessingThread() ()
No symbol table info available.
#7  0x0000000001b37830 in execute_native_thread_routine ()
No symbol table info available.
#8  0x00007fc9564e06aa in start_thread (arg=0x7fc9558c2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fc9558c2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502700402432, 5874792978760040640, 0, 140734797417775, 140502700403136, 140734797418552, -5902567582192140096, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#9  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

      Disassembly:

      => 0x19e0bf0 <__wt_split_stash_discard_all+48>:	cmpq   $0x0,(%rbx)
   0x19e0bf4 <__wt_split_stash_discard_all+52>:	je     0x19e0c01 <__wt_split_stash_discard_all+65>
   0x19e0bf6 <__wt_split_stash_discard_all+54>:	mov    %rbx,%rsi
   0x19e0bf9 <__wt_split_stash_discard_all+57>:	mov    %r14,%rdi
   0x19e0bfc <__wt_split_stash_discard_all+60>:	callq  0x1a47e50 <__wt_free_int>
   0x19e0c01 <__wt_split_stash_discard_all+65>:	add    $0x1,%r12
   0x19e0c05 <__wt_split_stash_discard_all+69>:	add    $0x18,%rbx
   0x19e0c09 <__wt_split_stash_discard_all+73>:	cmp    %r12,0x2e8(%r13)
   0x19e0c10 <__wt_split_stash_discard_all+80>:	ja     0x19e0bf0 <__wt_split_stash_discard_all+48>
   0x19e0c12 <__wt_split_stash_discard_all+82>:	mov    0x2e0(%r13),%rbx
   0x19e0c19 <__wt_split_stash_discard_all+89>:	test   %rbx,%rbx
   0x19e0c1c <__wt_split_stash_discard_all+92>:	je     0x19e0c2d <__wt_split_stash_discard_all+109>
   0x19e0c1e <__wt_split_stash_discard_all+94>:	lea    0x2e0(%r13),%rsi
   0x19e0c25 <__wt_split_stash_discard_all+101>:	mov    %r14,%rdi
   0x19e0c28 <__wt_split_stash_discard_all+104>:	callq  0x1a47e50 <__wt_free_int>
   0x19e0c2d <__wt_split_stash_discard_all+109>:	pop    %rbx

      Also a note: crashed primary was secondary a few hours before, but transition to primary automatically (ex-primary didn't crash, just stepped down from primary by itself)

        1. Disassembly
          1 kB
        2. mdiag.json.zip
          463 kB
        3. mongo_log_at_the_time_of_crash
          20 kB
        4. SegvAnalysis
          0.3 kB
        5. Stacktrace
          2 kB
        6. StacktraceTop
          0.2 kB
        7. ThreadStacktrace
          22 kB

            Assignee:
            david.hows David Hows
            Reporter:
            KarrotKake Jimmy [X]
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: