Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-25247

MongoDB primary node crashes after SIGINT (ctrl+c)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Duplicate
    • Affects Version/s: 3.2.6, 3.2.7, 3.2.8
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
    • Environment:
      Ubuntu 15.10, AMD64
    • Operating System:
      ALL
    • Linked BF Score:
      0

      Description

      After sending SIGINT to the primary node of the replica set (3 node dev setup: primary, secondary, arbiter) mongod crashed with SIGSEGV in __wt_split_stash_discard_all().

      Log from primary at the time of crash is in the attachment

      SegvAnalysis from Apport:

      Segfault happened at: 0x19e0bf0 <__wt_split_stash_discard_all+48>:	cmpq   $0x0,(%rbx)
      PC (0x019e0bf0) ok
      source "$0x0" ok
      destination "(%rbx)" (0x7fba414a9357) not located in a known VMA region (needed writable region)!
      Stack memory exhausted (SP below stack segment)
      

      Stacktrace:

      #0  0x00000000019e0bf0 in __wt_split_stash_discard_all ()
      No symbol table info available.
      #1  0x0000000001a078af in __wt_connection_close ()
      No symbol table info available.
      #2  0x00000000019fd1e0 in __conn_close ()
      No symbol table info available.
      #3  0x0000000001085db3 in mongo::WiredTigerKVEngine::cleanShutdown() ()
      No symbol table info available.
      #4  0x0000000000facdd8 in mongo::ServiceContextMongoD::shutdownGlobalStorageEngineCleanly() ()
      No symbol table info available.
      #5  0x0000000000cd1210 in mongo::exitCleanly(mongo::ExitCode) ()
      No symbol table info available.
      #6  0x000000000131b8c1 in mongo::(anonymous namespace)::signalProcessingThread() ()
      No symbol table info available.
      #7  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #8  0x00007fc9564e06aa in start_thread (arg=0x7fc9558c2700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc9558c2700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502700402432, 5874792978760040640, 0, 140734797417775, 140502700403136, 140734797418552, -5902567582192140096, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #9  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      

      Thread stacktrace:

      .
      Thread 15 (Thread 0x7fc94c0af700 (LWP 2551)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
      No symbol table info available.
      #2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
      No symbol table info available.
      #3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
      No symbol table info available.
      #4  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #5  0x00007fc9564e06aa in start_thread (arg=0x7fc94c0af700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94c0af700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502540941056, 5874792978760040640, 0, 140502582899919, 140502540941760, 34713112, -5902551138372976448, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 14 (Thread 0x7fc94b0ad700 (LWP 2553)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
      No symbol table info available.
      #2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
      No symbol table info available.
      #3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
      No symbol table info available.
      #4  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #5  0x00007fc9564e06aa in start_thread (arg=0x7fc94b0ad700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94b0ad700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502524155648, 5874792978760040640, 0, 140502582899919, 140502524156352, 34713112, -5902562134562996032, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 13 (Thread 0x7fc94f8b6700 (LWP 2544)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
      No symbol table info available.
      #2  0x0000000001273c48 in mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>::deadlineMonitorThread() ()
      No symbol table info available.
      #3  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #4  0x00007fc9564e06aa in start_thread (arg=0x7fc94f8b6700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94f8b6700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502599689984, 5874792978760040640, 0, 140734797414975, 140502599690688, 140734797417680, -5902554450329632576, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #5  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 12 (Thread 0x7fc9518ba700 (LWP 2540)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
      No locals.
      #1  0x0000000000bb77f9 in mongo::CondVarLockGrantNotification::wait(unsigned int) ()
      No symbol table info available.
      #2  0x0000000000bbb2c7 in mongo::LockerImpl<false>::lockComplete(mongo::ResourceId, mongo::LockMode, unsigned int, bool) ()
      No symbol table info available.
      #3  0x0000000000bb7e9c in mongo::LockerImpl<false>::lockGlobal(mongo::LockMode, unsigned int) ()
      No symbol table info available.
      #4  0x0000000000bb7b6d in mongo::LockerImpl<false>::restoreLockState(mongo::Locker::LockSnapshot const&) ()
      No symbol table info available.
      #5  0x0000000001092240 in mongo::WiredTigerRecordStore::yieldAndAwaitOplogDeletionRequest(mongo::OperationContext*) ()
      No symbol table info available.
      #6  0x00000000010981a3 in mongo::(anonymous namespace)::WiredTigerRecordStoreThread::_deleteExcessDocuments() ()
      No symbol table info available.
      #7  0x0000000001098718 in mongo::(anonymous namespace)::WiredTigerRecordStoreThread::run() ()
      No symbol table info available.
      #8  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
      No symbol table info available.
      #9  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #10 0x00007fc9564e06aa in start_thread (arg=0x7fc9518ba700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc9518ba700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502633260800, 5874792978760040640, 0, 140734797411743, 140502633261504, 140734797412240, -5902576434119737152, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #11 0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 11 (Thread 0x7fc94d8b2700 (LWP 2548)):
      #0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      No locals.
      #1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
              ts = {tv_sec = 8, tv_nsec = 470999445}
              set = {__val = {65536, 0 <repeats 15 times>}}
              oset = {__val = {8405507, 140502566115200, 140502566115024, 67575832, 140827780, 140827776, 0, 0, 0, 69083232, 67951712, 140501265154048, 129257520, 129258768, 129257520, 2}}
              result = <optimized out>
      #2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
      No symbol table info available.
      #3  0x0000000001326aee in mongo::sleepsecs(int) ()
      No symbol table info available.
      #4  0x00000000010a917b in mongo::TTLMonitor::run() ()
      No symbol table info available.
      #5  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
      No symbol table info available.
      #6  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #7  0x00007fc9564e06aa in start_thread (arg=0x7fc94d8b2700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94d8b2700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502566119168, 5874792978760040640, 0, 140734797414959, 140502566119872, 140734797417680, -5902550050135637824, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #8  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 10 (Thread 0x7fc94c8b0700 (LWP 2550)):
      #0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      No locals.
      #1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
              ts = {tv_sec = 2, tv_nsec = 813136127}
              set = {__val = {65536, 0 <repeats 15 times>}}
              oset = {__val = {8405507, 65122808, 140502549329567, 140502549329632, 140502549329600, 10190813, 140502549329632, 14, 65122560, 0, 140502549329664, 140502549329904, 140502549329856, 11499820, 67783952, 140502549329632}}
              result = <optimized out>
      #2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
      No symbol table info available.
      #3  0x0000000001326aee in mongo::sleepsecs(int) ()
      No symbol table info available.
      #4  0x0000000000b14fd8 in mongo::ClientCursorMonitor::run() ()
      No symbol table info available.
      #5  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
      No symbol table info available.
      #6  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #7  0x00007fc9564e06aa in start_thread (arg=0x7fc94c8b0700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94c8b0700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502549333760, 5874792978760040640, 0, 140734797414959, 140502549334464, 140734797417680, -5902552237347733312, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #8  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 9 (Thread 0x7fc94b8ae700 (LWP 2552)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
      No locals.
      #1  0x00000000012abb9e in mongo::(anonymous namespace)::PeriodicTaskRunner::run() ()
      No symbol table info available.
      #2  0x00000000012ab0d0 in mongo::BackgroundJob::jobBody() ()
      No symbol table info available.
      #3  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #4  0x00007fc9564e06aa in start_thread (arg=0x7fc94b8ae700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94b8ae700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502532548352, 5874792978760040640, 0, 140734797414911, 140502532549056, 140734797417680, -5902563233537752896, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #5  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 8 (Thread 0x7fc94a8ac700 (LWP 2554)):
      #0  0x00007fc9561d9e7d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
      No locals.
      #1  0x00007fc9561d9d14 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
              ts = {tv_sec = 8, tv_nsec = 147663717}
              set = {__val = {65536, 0 <repeats 15 times>}}
              oset = {__val = {8405507, 0 <repeats 15 times>}}
              result = <optimized out>
      #2  0x0000000001b377c9 in std::this_thread::__sleep_for(std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ()
      No symbol table info available.
      #3  0x0000000001326aee in mongo::sleepsecs(int) ()
      No symbol table info available.
      #4  0x00000000012bf49e in mongo::HostnameCanonicalizationWorker::_doWork() ()
      No symbol table info available.
      #5  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #6  0x00007fc9564e06aa in start_thread (arg=0x7fc94a8ac700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94a8ac700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502515762944, 5874792978760040640, 0, 140734797414943, 140502515763648, 140734797417680, -5902565429339782976, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #7  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 7 (Thread 0x7fc94f0b5700 (LWP 2545)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
      No locals.
      #1  0x0000000000e56b43 in mongo::RangeDeleter::doWork() ()
      No symbol table info available.
      #2  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #3  0x00007fc9564e06aa in start_thread (arg=0x7fc94f0b5700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94f0b5700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502591297280, 5874792978760040640, 0, 140734797415039, 140502591297984, 140734797417680, -5902553351354875712, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #4  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 6 (Thread 0x7fc94d0b1700 (LWP 2549)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001b363dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
      No symbol table info available.
      #2  0x00000000012b1b06 in mongo::ThreadPool::_consumeTasks() ()
      No symbol table info available.
      #3  0x00000000012b22b0 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::string const&) ()
      No symbol table info available.
      #4  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #5  0x00007fc9564e06aa in start_thread (arg=0x7fc94d0b1700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc94d0b1700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502557726464, 5874792978760040640, 0, 140502582899919, 140502557727168, 34713112, -5902548951160880960, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 5 (Thread 0x7fc946e99700 (LWP 2564)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001338af0 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) ()
      No symbol table info available.
      #2  0x0000000001338da1 in asio::detail::scheduler::run(std::error_code&) ()
      No symbol table info available.
      #3  0x000000000133cf3f in asio::io_service::run() ()
      No symbol table info available.
      #4  0x00000000013306e0 in asio_detail_posix_thread_function ()
      No symbol table info available.
      #5  0x00007fc9564e06aa in start_thread (arg=0x7fc946e99700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc946e99700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502454867712, 5874792978760040640, 0, 140502574506895, 140502454868416, 140502574507232, -5902538209447673664, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 4 (Thread 0x7fc9578d0cc0 (LWP 2530)):
      #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
      No locals.
      #1  0x00007fc9564e2cfd in __GI___pthread_mutex_lock (mutex=0x20b4340 <mongo::shutdownLock>) at ../nptl/pthread_mutex_lock.c:80
              __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
              type = 0
              id = <optimized out>
      #2  0x0000000000cd0eb5 in mongo::exitCleanly(mongo::ExitCode) ()
      No symbol table info available.
      #3  0x000000000096e054 in main ()
      No symbol table info available.
      .
      Thread 3 (Thread 0x7fc934270700 (LWP 24993)):
      #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
      No locals.
      #1  0x0000000000ae5e77 in mongo::CappedInsertNotifier::_wait(std::unique_lock<std::mutex>&, unsigned long, std::chrono::duration<long, std::ratio<1l, 1000000l> >) const ()
      No symbol table info available.
      #2  0x0000000000ae8699 in mongo::CappedInsertNotifier::wait(unsigned long, std::chrono::duration<long, std::ratio<1l, 1000000l> >) const ()
      No symbol table info available.
      #3  0x0000000000b5389f in mongo::GetMoreCmd::run(mongo::OperationContext*, std::string const&, mongo::BSONObj&, int, std::string&, mongo::BSONObjBuilder&) ()
      No symbol table info available.
      #4  0x0000000000bc7f93 in mongo::Command::run(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
      No symbol table info available.
      #5  0x0000000000bc8e24 in mongo::Command::execCommand(mongo::OperationContext*, mongo::Command*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
      No symbol table info available.
      #6  0x0000000000b24ce0 in mongo::runCommands(mongo::OperationContext*, mongo::rpc::RequestInterface const&, mongo::rpc::ReplyBuilderInterface*) ()
      No symbol table info available.
      #7  0x0000000000cd6a15 in mongo::assembleResponse(mongo::OperationContext*, mongo::Message&, mongo::DbResponse&, mongo::HostAndPort const&) ()
      No symbol table info available.
      #8  0x00000000009b937c in mongo::MyMessageHandler::process(mongo::Message&, mongo::AbstractMessagingPort*) ()
      No symbol table info available.
      #9  0x00000000012c9645 in mongo::PortMessageServer::handleIncomingMsg(void*) ()
      No symbol table info available.
      #10 0x00007fc9564e06aa in start_thread (arg=0x7fc934270700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc934270700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502140126976, 5874792978760040640, 0, 140734797414223, 140502140127680, 8388608, -5902779458297560896, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #11 0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 2 (Thread 0x7fc935575700 (LWP 2672)):
      #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
      No locals.
      #1  0x0000000001338af0 in asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) ()
      No symbol table info available.
      #2  0x0000000001338da1 in asio::detail::scheduler::run(std::error_code&) ()
      No symbol table info available.
      #3  0x000000000133cf3f in asio::io_service::run() ()
      No symbol table info available.
      #4  0x00000000013306e0 in asio_detail_posix_thread_function ()
      No symbol table info available.
      #5  0x00007fc9564e06aa in start_thread (arg=0x7fc935575700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc935575700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502160070400, 5874792978760040640, 0, 140502471648143, 8388608, 124960064, -5902776861452959552, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #6  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      .
      Thread 1 (Thread 0x7fc9558c2700 (LWP 2531)):
      #0  0x00000000019e0bf0 in __wt_split_stash_discard_all ()
      No symbol table info available.
      #1  0x0000000001a078af in __wt_connection_close ()
      No symbol table info available.
      #2  0x00000000019fd1e0 in __conn_close ()
      No symbol table info available.
      #3  0x0000000001085db3 in mongo::WiredTigerKVEngine::cleanShutdown() ()
      No symbol table info available.
      #4  0x0000000000facdd8 in mongo::ServiceContextMongoD::shutdownGlobalStorageEngineCleanly() ()
      No symbol table info available.
      #5  0x0000000000cd1210 in mongo::exitCleanly(mongo::ExitCode) ()
      No symbol table info available.
      #6  0x000000000131b8c1 in mongo::(anonymous namespace)::signalProcessingThread() ()
      No symbol table info available.
      #7  0x0000000001b37830 in execute_native_thread_routine ()
      No symbol table info available.
      #8  0x00007fc9564e06aa in start_thread (arg=0x7fc9558c2700) at pthread_create.c:333
              __res = <optimized out>
              pd = 0x7fc9558c2700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140502700402432, 5874792978760040640, 0, 140734797417775, 140502700403136, 140734797418552, -5902567582192140096, -5902572547200155456}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
              __PRETTY_FUNCTION__ = "start_thread"
      #9  0x00007fc95621613d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      No locals.
      

      Disassembly:

      => 0x19e0bf0 <__wt_split_stash_discard_all+48>:	cmpq   $0x0,(%rbx)
         0x19e0bf4 <__wt_split_stash_discard_all+52>:	je     0x19e0c01 <__wt_split_stash_discard_all+65>
         0x19e0bf6 <__wt_split_stash_discard_all+54>:	mov    %rbx,%rsi
         0x19e0bf9 <__wt_split_stash_discard_all+57>:	mov    %r14,%rdi
         0x19e0bfc <__wt_split_stash_discard_all+60>:	callq  0x1a47e50 <__wt_free_int>
         0x19e0c01 <__wt_split_stash_discard_all+65>:	add    $0x1,%r12
         0x19e0c05 <__wt_split_stash_discard_all+69>:	add    $0x18,%rbx
         0x19e0c09 <__wt_split_stash_discard_all+73>:	cmp    %r12,0x2e8(%r13)
         0x19e0c10 <__wt_split_stash_discard_all+80>:	ja     0x19e0bf0 <__wt_split_stash_discard_all+48>
         0x19e0c12 <__wt_split_stash_discard_all+82>:	mov    0x2e0(%r13),%rbx
         0x19e0c19 <__wt_split_stash_discard_all+89>:	test   %rbx,%rbx
         0x19e0c1c <__wt_split_stash_discard_all+92>:	je     0x19e0c2d <__wt_split_stash_discard_all+109>
         0x19e0c1e <__wt_split_stash_discard_all+94>:	lea    0x2e0(%r13),%rsi
         0x19e0c25 <__wt_split_stash_discard_all+101>:	mov    %r14,%rdi
         0x19e0c28 <__wt_split_stash_discard_all+104>:	callq  0x1a47e50 <__wt_free_int>
         0x19e0c2d <__wt_split_stash_discard_all+109>:	pop    %rbx
      

      Also a note: crashed primary was secondary a few hours before, but transition to primary automatically (ex-primary didn't crash, just stepped down from primary by itself)

        Attachments

        1. Disassembly
          1 kB
        2. mdiag.json.zip
          463 kB
        3. mongo_log_at_the_time_of_crash
          20 kB
        4. SegvAnalysis
          0.3 kB
        5. Stacktrace
          2 kB
        6. StacktraceTop
          0.2 kB
        7. ThreadStacktrace
          22 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: