WT recovery rollback to stable checkpoint hangs on writing checkpoint log record.
As described in BF-16822, server restart hangs during recovery. Following is the call stack where it waits
Thread 1 (Thread 0x7ffff7fd6180 (LWP 76029)):
#0 __log_wait_for_earlier_slot () at src/third_party/wiredtiger/src/log/log.c:155
#1 0x0000555556849c4b in __wt_log_release () at src/third_party/wiredtiger/src/log/log.c:1909
#2 0x000055555684c0bf in __log_write_internal (flags=8, lsnp=0x0, record=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/log/log.c:2668
#3 __wt_log_write () at src/third_party/wiredtiger/src/log/log.c:2551
#4 0x00005555568a9bcd in __wt_txn_checkpoint_log (session=0x55555a851108, full=<optimized out>, flags=<optimized out>, lsnp=0x0) at src/third_party/wiredtiger/src/txn/txn_log.c:543
#5 0x00005555568a7177 in __txn_checkpoint (cfg=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1037
#6 __txn_checkpoint_wrapper () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1076
#7 0x00005555568a7f39 in __wt_txn_checkpoint () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1133
#8 0x00005555561c327c in __session_checkpoint (wt_session=0x55555a851108, config=<optimized out>) at src/third_party/wiredtiger/src/session/session_api.c:1979
#9 0x00005555568b33c0 in __wt_rollback_to_stable () at src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c:1127
#10 0x00005555568abe09 in __wt_txn_recover () at src/third_party/wiredtiger/src/txn/txn_recover.c:758
#11 0x000055555681c6a7 in __wt_connection_workers (session=0x55555a84f000, cfg=0x7fffffffac20) at src/third_party/wiredtiger/src/conn/conn_open.c:211
#12 0x000055555681834d in wiredtiger_open () at src/third_party/wiredtiger/src/conn/conn_api.c:2677
#13 0x00005555567d137a in mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/basic_string.h:2281
#14 0x00005555567d6bf8 in mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool, bool, bool) ()
at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:908
#15 0x000055555679dc28 in mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const ()
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/unique_ptr.h:342
---Type <return> to continue, or q <return> to quit---
#16 0x0000555556ef13d5 in mongo::initializeStorageEngine(mongo::ServiceContext*, mongo::StorageEngineInitFlags) () at src/third_party/boost-1.70.0/boost/optional/optional.hpp:1448
#17 0x0000555556543796 in mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) [clone .isra.1294] () at src/mongo/db/db.cpp:357
#18 0x0000555556547199 in mongo::(anonymous namespace)::initAndListen (listenPort=<optimized out>, service=0x55555a642400) at src/mongo/db/db.cpp:1319
#19 mongo::(anonymous namespace)::mongoDbMain(int, char**, char**) () at src/mongo/db/db.cpp:1319
#20 0x00005555564a9269 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at src/mongo/db/db.cpp:1340
(gdb) c
The code is looping in that function to let the log->write_lsn to match with lot->slot_release_lsn. It seems there is no progress with log->write_lsn lead to wait forever. Values of write_lsn and slot_release_lsn are:
(gdb) p log->write_lsn
$2 = {l = {offset = 256, file = 694}, file_offset = 2980707303680}
(gdb) p slot->slot_release_lsn
$3 = {l = {offset = 1101824, file = 694}, file_offset = 2980708405248}
Following are the call stacks of all the threads that are running in the system when this issue happened.
Thread 9 (Thread 0x7fffeb7bb700 (LWP 76040)):
#0 0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
#2 0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a850c50, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffeb7ba847)
at src/third_party/wiredtiger/src/support/cond_auto.c:61
#3 0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a850c50, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
at src/third_party/wiredtiger/src/support/cond_auto.c:80
#4 0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
#5 0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
#6 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fffebfbc700 (LWP 76039)):
#0 0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
#2 0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a850798, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffebfbb847)
at src/third_party/wiredtiger/src/support/cond_auto.c:61
#3 0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a850798, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
at src/third_party/wiredtiger/src/support/cond_auto.c:80
#4 0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
#5 0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
#6 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fffec7bd700 (LWP 76038)):
#0 0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
#2 0x00005555568907d6 in __wt_cond_auto_wait_signal (session=session@entry=0x55555a8502e0, cond=0x55555a7cc380, progress=progress@entry=false, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fffec7bc847)
at src/third_party/wiredtiger/src/support/cond_auto.c:61
#3 0x0000555556890823 in __wt_cond_auto_wait (session=session@entry=0x55555a8502e0, cond=<optimized out>, progress=progress@entry=false, run_func=run_func@entry=0x0)
at src/third_party/wiredtiger/src/support/cond_auto.c:80
#4 0x000055555683d63c in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:307
#5 0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
#6 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fffecfbe700 (LWP 76037)):
#0 0x00007ffff495359e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555556864499 in __wt_cond_wait_signal () at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:115
#2 0x0000555556839b52 in __wt_cond_wait (run_func=0x0, usecs=10000, cond=<optimized out>, session=0x55555a84fe28) at src/third_party/wiredtiger/src/include/misc.i:19
#3 __evict_lru_pages (session=session@entry=0x55555a84fe28, is_server=is_server@entry=false) at src/third_party/wiredtiger/src/evict/evict_lru.c:1123
#4 0x000055555683b979 in __wt_evict_thread_run () at src/third_party/wiredtiger/src/evict/evict_lru.c:311
#5 0x000055555689f1c9 in __thread_run () at src/third_party/wiredtiger/src/support/thread_group.c:31
#6 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#7 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fffed7bf700 (LWP 76036)):
#0 0x00007ffff495361e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000555558207d97 in __gthread_cond_timedwait (__abs_timeout=0x7fffed7be900, __mutex=<optimized out>, __cond=0x55555a7c62d0)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/x86_64-mongodb-linux/bits/gthr-default.h:871
#2 std::condition_variable::__wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
---Type <return> to continue, or q <return> to quit---
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:178
#3 std::condition_variable::wait_until<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:106
#4 std::_V2::condition_variable_any::wait_until<mongo::BasicLockableAdapter, std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (__atime=..., __lock=..., this=0x55555a7c62d0)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/condition_variable:286
#5 mongo::Waitable::wait_until<mongo::BasicLockableAdapter> (timeout_time=..., lk=..., cv=..., clkSource=0x55555a121f10, waitable=<optimized out>) at src/mongo/util/waitable.h:102
#6 mongo::ClockSource::waitForConditionUntil(mongo::stdx::condition_variable&, mongo::BasicLockableAdapter, mongo::Date_t, mongo::Waitable*) () at src/mongo/util/clock_source.cpp:48
#7 0x0000555556bc7ad2 in mongo::ClockSource::waitForConditionUntil<std::unique_lock<mongo::latch_detail::Latch>, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> mutable::<lambda()> > (waitable=0x0,
pred=..., deadline=..., m=..., cv=..., this=0x55555a121f10) at src/mongo/util/lockable_adapter.h:50
#8 mongo::PeriodicRunnerImpl::PeriodicJobImpl::<lambda()>::operator() (__closure=<optimized out>) at src/mongo/util/periodic_runner_impl.cpp:100
#9 std::__invoke_impl<void, mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#10 std::__invoke<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()> > (__fn=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#11 std::__apply_impl<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
#12 std::apply<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
#13 mongo::stdx::thread::<lambda()>::operator() (this=<optimized out>) at src/mongo/stdx/thread.h:186
#14 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#15 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#16 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=<optimized out>)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
#17 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=<optimized out>)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
#18 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}, , 0>(mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
#19 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#20 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#21 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fffedfc0700 (LWP 76035)):
#0 0x00007ffff4953277 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00005555584e170c in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x55555a7cdb70)
at /data/mci/4aa8627b70212eb05b788e4bc6eb2ee3/toolchain-builder/tmp/build-gcc-v3.sh-uyz/build/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864
#2 std::condition_variable::wait (this=this@entry=0x55555a7cdb70, __lock=...) at ../../../../../src/combined/libstdc++-v3/src/c++11/condition_variable.cc:53
#3 0x0000555557e63c43 in std::_V2::condition_variable_any::wait<std::unique_lock<mongo::latch_detail::Latch> > (__lock=..., this=0x55555a7cdb70)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/std_mutex.h:238
#4 mongo::ThreadPool::_consumeTasks() () at src/mongo/util/concurrency/thread_pool.cpp:341
#5 0x0000555557e645ea in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
at src/mongo/util/concurrency/thread_pool.cpp:273
#6 0x0000555557e64930 in mongo::ThreadPool::<lambda()>::operator() (__closure=0x55555a6caf50) at src/mongo/util/concurrency/thread_pool.cpp:452
#7 std::__invoke_impl<void, mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()> > (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#8 std::__invoke<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#9 std::__apply_impl<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
---Type <return> to continue, or q <return> to quit---
#10 std::apply<mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
#11 mongo::stdx::thread::<lambda()>::operator() (this=0x55555a6caf48) at src/mongo/stdx/thread.h:186
#12 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#13 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#14 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=0x55555a6caf48)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
#15 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::ThreadPool::_startWorkerThread_inlock()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=0x55555a6caf48)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
#16 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::_startWorkerThread_inlock()::{lambda()#4}, , 0>(mongo::ThreadPool::_startWorkerThread_inlock()::{lambda()#4})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
#17 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#18 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#19 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fffee7c1700 (LWP 76034)):
#0 0x00007ffff4953277 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00005555584e170c in __gthread_cond_wait (__mutex=<optimized out>, __cond=__cond@entry=0x55555a790c50)
at /data/mci/4aa8627b70212eb05b788e4bc6eb2ee3/toolchain-builder/tmp/build-gcc-v3.sh-uyz/build/x86_64-mongodb-linux/libstdc++-v3/include/x86_64-mongodb-linux/bits/gthr-default.h:864
#2 std::condition_variable::wait (this=this@entry=0x55555a790c50, __lock=...) at ../../../../../src/combined/libstdc++-v3/src/c++11/condition_variable.cc:53
#3 0x00005555582070b7 in std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> > (__lock=..., this=0x55555a790c50)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/std_mutex.h:238
#4 mongo::BackgroundThreadClockSource::<lambda()>::operator() (__closure=0x55555a7b1e30) at src/mongo/util/background_thread_clock_source.cpp:188
#5 std::__invoke_impl<void, mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()> > (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#6 std::__invoke<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()> > (__fn=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#7 std::__apply_impl<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
#8 std::apply<mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>, std::tuple<> > (__t=..., __f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
#9 mongo::stdx::thread::<lambda()>::operator() (this=0x55555a7b1e28) at src/mongo/stdx/thread.h:186
#10 std::__invoke_impl<void, mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#11 std::__invoke<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > (__fn=...) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#12 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::_M_invoke<0> (this=0x55555a7b1e28)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
#13 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread(Function, Args&& ...) [with Function = mongo::BackgroundThreadClockSource::_startTimerThread()::<lambda()>; Args = {}; typename std::enable_if<(! is_same_v<mongo::stdx::thread, typename std::decay<_Tp>::type>), int>::type <anonymous> = 0]::<lambda()> > >::operator() (this=0x55555a7b1e28)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
#14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundThreadClockSource::_startTimerThread()::{lambda()#1}, , 0>(mongo::BackgroundThreadClockSource::_startTimerThread()::{lambda()#1})::{lambda()#1}> > >::_M_run() () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
#15 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#16 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#17 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 2 (Thread 0x7fffeefc2700 (LWP 76033)):
#0 0x00007ffff45cf9f1 in sigwaitinfo () from /lib64/libc.so.6
#1 0x0000555556dd37a9 in mongo::(anonymous namespace)::waitForSignal (result=0x7fffeefc17b0, sigset=...) at src/mongo/util/signal_handlers.cpp:177
#2 mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus) () at src/mongo/util/signal_handlers.cpp:295
#3 0x0000555556dd3ff5 in std::__invoke_impl<void, void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:89
#4 std::__invoke<void (*)(mongo::LogFileStatus), mongo::LogFileStatus> (__fn=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#5 std::__apply_impl<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus>, 0ul> (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>, __t=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1678
#6 std::apply<void (*)(mongo::LogFileStatus), std::tuple<mongo::LogFileStatus> > (__f=@0x55555a7b1d50: 0x555556dd35a0 <mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus)>, __t=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/tuple:1687
#7 mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}::operator()() (this=0x55555a7b1d48)
at src/mongo/stdx/thread.h:186
#8 std::__invoke_impl<void, mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}>(std::__invoke_other, mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}&&) (__f=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:60
#9 std::__invoke<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}>(mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}&&) (__fn=...)
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/invoke.h:95
#10 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x55555a7b1d48) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:234
#11 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> >::operator()() (
this=0x55555a7b1d48) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:243
#12 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<void (*)(mongo::LogFileStatus), mongo::LogFileStatus&, 0>(void (*)(mongo::LogFileStatus), mongo::LogFileStatus&)::{lambda()#1}> > >::_M_run() (this=0x55555a7b1d40) at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/thread:186
#13 0x00005555584e48af in execute_native_thread_routine () at ../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80
#14 0x00007ffff494d40b in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff4687e7f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ffff7fd6180 (LWP 76029)):
#0 __log_wait_for_earlier_slot () at src/third_party/wiredtiger/src/log/log.c:155
#1 0x0000555556849c4b in __wt_log_release () at src/third_party/wiredtiger/src/log/log.c:1909
#2 0x000055555684c0bf in __log_write_internal (flags=8, lsnp=0x0, record=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/log/log.c:2668
#3 __wt_log_write () at src/third_party/wiredtiger/src/log/log.c:2551
#4 0x00005555568a9bcd in __wt_txn_checkpoint_log (session=0x55555a851108, full=<optimized out>, flags=<optimized out>, lsnp=0x0) at src/third_party/wiredtiger/src/txn/txn_log.c:543
#5 0x00005555568a7177 in __txn_checkpoint (cfg=<optimized out>, session=0x55555a851108) at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1037
#6 __txn_checkpoint_wrapper () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1076
#7 0x00005555568a7f39 in __wt_txn_checkpoint () at src/third_party/wiredtiger/src/txn/txn_ckpt.c:1133
#8 0x00005555561c327c in __session_checkpoint (wt_session=0x55555a851108, config=<optimized out>) at src/third_party/wiredtiger/src/session/session_api.c:1979
#9 0x00005555568b33c0 in __wt_rollback_to_stable () at src/third_party/wiredtiger/src/txn/txn_rollback_to_stable.c:1127
#10 0x00005555568abe09 in __wt_txn_recover () at src/third_party/wiredtiger/src/txn/txn_recover.c:758
#11 0x000055555681c6a7 in __wt_connection_workers (session=0x55555a84f000, cfg=0x7fffffffac20) at src/third_party/wiredtiger/src/conn/conn_open.c:211
#12 0x000055555681834d in wiredtiger_open () at src/third_party/wiredtiger/src/conn/conn_api.c:2677
#13 0x00005555567d137a in mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/basic_string.h:2281
#14 0x00005555567d6bf8 in mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool, bool, bool) ()
at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:908
#15 0x000055555679dc28 in mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const ()
at /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.qpt/include/c++/8.2.0/bits/unique_ptr.h:342
---Type <return> to continue, or q <return> to quit---
#16 0x0000555556ef13d5 in mongo::initializeStorageEngine(mongo::ServiceContext*, mongo::StorageEngineInitFlags) () at src/third_party/boost-1.70.0/boost/optional/optional.hpp:1448
#17 0x0000555556543796 in mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) [clone .isra.1294] () at src/mongo/db/db.cpp:357
#18 0x0000555556547199 in mongo::(anonymous namespace)::initAndListen (listenPort=<optimized out>, service=0x55555a642400) at src/mongo/db/db.cpp:1319
#19 mongo::(anonymous namespace)::mongoDbMain(int, char**, char**) () at src/mongo/db/db.cpp:1319
#20 0x00005555564a9269 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at src/mongo/db/db.cpp:1340
(gdb) c
Continuing.
- is related to
-
WT-5996 Review WT_SESSION_NO_LOGGING and if other flags should be retained when calling RTS
-
- Backlog
-