Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-14199

Fix deadlock in live restore code

    • Storage Engines
    • 3
    • StorEng - 2025-02-28, StorEng - 2025-03-14

      See task: https://evergreen.mongodb.com/task_log_raw/wiredtiger_live_restore_ubuntu2004_live_restore_long_9825f6a8cc3b413f3e5041b742d511548eab8120_25_02_20_22_02_13/0?type=T

      Stacks:

      [2025/02/20 17:42:41.542] Thread 18 (Thread 0x7fc8057fa700 (LWP 23644) "checkpoint-clea"):
       [2025/02/20 17:42:41.542] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109e308, cond=0xf707a0, usecs=usecs@entry=5000000, run_func=run_func@entry=0x7fc84d7a2db8 <__checkpoint_cleanup_run_chk>, signalled=signalled@entry=0x7fc8057f8b0f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.542] #2  0x00007fc84d7a91f2 in __checkpoint_cleanup (arg=0x109e308) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/btree/bt_sync_obsolete.c:761
       [2025/02/20 17:42:41.542] #3  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #4  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.542] Thread 17 (Thread 0x7fc805ffb700 (LWP 23643) "compact-server"):
       [2025/02/20 17:42:41.542] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109dc80, cond=0xcb6d00, usecs=10000000, run_func=run_func@entry=0x7fc84d7fb6ab <__background_compact_server_run_chk>, signalled=signalled@entry=0x7fc805ff9ae7) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.542] #2  0x00007fc84d7fcaa0 in __wt_cond_wait (run_func=0x7fc84d7fb6ab <__background_compact_server_run_chk>, usecs=<optimized out>, cond=<optimized out>, session=0x109dc80) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.542] #3  __background_compact_server (arg=0x109dc80) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/conn/conn_compact.c:580
       [2025/02/20 17:42:41.542] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.542] Thread 16 (Thread 0x7fc8067fc700 (LWP 23642) "sweep-server"):
       [2025/02/20 17:42:41.542] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109d5f8, cond=0x205a8d0, usecs=usecs@entry=10000000, run_func=run_func@entry=0x7fc84d8089f0 <__sweep_server_run_chk>, signalled=signalled@entry=0x7fc8067fab0b) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.542] #2  0x00007fc84d8099fa in __sweep_server (arg=0x109d5f8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/conn/conn_sweep.c:380
       [2025/02/20 17:42:41.542] #3  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.542] #4  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.542] Thread 15 (Thread 0x7fc806ffd700 (LWP 23641) "eviction-ser 8"):
       [2025/02/20 17:42:41.543] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.543] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109cf70, cond=0x1dcb8d0, usecs=usecs@entry=10000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc806ffb720) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.543] #2  0x00007fc84d97af85 in __wt_cond_wait (run_func=<optimized out>, usecs=10000000, cond=<optimized out>, session=0x109cf70) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.543] #3  __thread_run (arg=0x2563c20) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:29
       [2025/02/20 17:42:41.543] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.543] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.543] Thread 14 (Thread 0x7fc8077fe700 (LWP 23640) "eviction-ser 7"):
       [2025/02/20 17:42:41.543] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.543] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109c8e8, cond=cond@entry=0xd213b0, usecs=1000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc8077fc69f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.544] #2  0x00007fc84d95e025 in __wt_cond_auto_wait_signal (session=session@entry=0x109c8e8, cond=0xd213b0, progress=<optimized out>, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fc8077fc69f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:70
       [2025/02/20 17:42:41.544] #3  0x00007fc84d95e185 in __wt_cond_auto_wait (session=session@entry=0x109c8e8, cond=<optimized out>, progress=<optimized out>, run_func=run_func@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:89
       [2025/02/20 17:42:41.544] #4  0x00007fc84d882d3d in __evict_thread_run (session=0x109c8e8, thread=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/evict/evict_lru.c:349
       [2025/02/20 17:42:41.544] #5  0x00007fc84d97af5a in __thread_run (arg=0x9f54f30) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:31
       [2025/02/20 17:42:41.545] #6  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #7  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.545] Thread 13 (Thread 0x7fc807fff700 (LWP 23639) "eviction-ser 6"):
       [2025/02/20 17:42:41.545] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109c260, cond=0x2c52390, usecs=usecs@entry=10000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc807ffd720) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.545] #2  0x00007fc84d97af85 in __wt_cond_wait (run_func=<optimized out>, usecs=10000000, cond=<optimized out>, session=0x109c260) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.545] #3  __thread_run (arg=0x12b13b0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:29
       [2025/02/20 17:42:41.545] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.545] Thread 12 (Thread 0x7fc834ff9700 (LWP 23638) "eviction-ser 5"):
       [2025/02/20 17:42:41.545] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109bbd8, cond=cond@entry=0xd213b0, usecs=1000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc834ff769f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.545] #2  0x00007fc84d95e025 in __wt_cond_auto_wait_signal (session=session@entry=0x109bbd8, cond=0xd213b0, progress=<optimized out>, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fc834ff769f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:70
       [2025/02/20 17:42:41.545] #3  0x00007fc84d95e185 in __wt_cond_auto_wait (session=session@entry=0x109bbd8, cond=<optimized out>, progress=<optimized out>, run_func=run_func@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:89
       [2025/02/20 17:42:41.545] #4  0x00007fc84d882d3d in __evict_thread_run (session=0x109bbd8, thread=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/evict/evict_lru.c:349
       [2025/02/20 17:42:41.545] #5  0x00007fc84d97af5a in __thread_run (arg=0x1163450) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:31
       [2025/02/20 17:42:41.545] #6  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #7  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.545] Thread 11 (Thread 0x7fc8357fa700 (LWP 23637) "eviction-ser 4"):
       [2025/02/20 17:42:41.545] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.545] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109b550, cond=0x47ae4f0, usecs=usecs@entry=10000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc8357f8720) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.545] #2  0x00007fc84d97af85 in __wt_cond_wait (run_func=<optimized out>, usecs=10000000, cond=<optimized out>, session=0x109b550) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.583] #3  __thread_run (arg=0x2bbd6c0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:29
       [2025/02/20 17:42:41.583] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.583] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.583] Thread 10 (Thread 0x7fc835ffb700 (LWP 23636) "eviction-ser 3"):
       [2025/02/20 17:42:41.583] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.583] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109aec8, cond=0xc22130, usecs=usecs@entry=10000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc835ff9720) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.583] #2  0x00007fc84d97af85 in __wt_cond_wait (run_func=<optimized out>, usecs=10000000, cond=<optimized out>, session=0x109aec8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.583] #3  __thread_run (arg=0x29c7290) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:29
       [2025/02/20 17:42:41.583] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.583] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.583] Thread 9 (Thread 0x7fc8367fc700 (LWP 23635) "eviction-ser 2"):
       [2025/02/20 17:42:41.583] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.583] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109a840, cond=0x2bbd630, usecs=usecs@entry=10000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc8367fa720) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.584] #2  0x00007fc84d97af85 in __wt_cond_wait (run_func=<optimized out>, usecs=10000000, cond=<optimized out>, session=0x109a840) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.584] #3  __thread_run (arg=0xc22380) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:29
       [2025/02/20 17:42:41.584] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.584] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.584] Thread 8 (Thread 0x7fc836ffd700 (LWP 23634) "eviction-ser 1"):
       [2025/02/20 17:42:41.584] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.584] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x109a1b8, cond=cond@entry=0xd213b0, usecs=1000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc836ffb69f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.584] #2  0x00007fc84d95e025 in __wt_cond_auto_wait_signal (session=session@entry=0x109a1b8, cond=0xd213b0, progress=<optimized out>, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fc836ffb69f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:70
       [2025/02/20 17:42:41.584] #3  0x00007fc84d95e185 in __wt_cond_auto_wait (session=session@entry=0x109a1b8, cond=<optimized out>, progress=<optimized out>, run_func=run_func@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:89
       [2025/02/20 17:42:41.584] #4  0x00007fc84d882d3d in __evict_thread_run (session=0x109a1b8, thread=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/evict/evict_lru.c:349
       [2025/02/20 17:42:41.584] #5  0x00007fc84d97af5a in __thread_run (arg=0x5db1590) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:31
       [2025/02/20 17:42:41.584] #6  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.584] #7  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.584] Thread 7 (Thread 0x7fc8377fe700 (LWP 23633) "log-server"):
       [2025/02/20 17:42:41.584] #0  0x00007fc84d6a6170 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.584] #1  0x00007fc84d69e0a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.584] #2  0x00007fc84d8b9910 in __wt_spin_lock (t=0x5606b18, session=0x1099b30) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/mutex_inline.h:192
       [2025/02/20 17:42:41.584] #3  __wt_spin_lock_track (t=0x5606b18, session=0x1099b30) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/mutex_inline.h:391
       [2025/02/20 17:42:41.653] #4  __wti_log_slot_switch (session=0x1099b30, myslot=myslot@entry=0x7fc8377fcaa0, retry=retry@entry=false, forced=forced@entry=true, did_work=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_slot.c:411
       [2025/02/20 17:42:41.653] #5  0x00007fc84d8a4cb2 in __wti_log_force_write (session=session@entry=0x1099b30, retry=retry@entry=false, did_work=did_work@entry=0x7fc8377fcaff) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2476
       [2025/02/20 17:42:41.653] #6  0x00007fc84d8b507c in __log_server (arg=0x1099b30) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_mgr.c:941
       [2025/02/20 17:42:41.654] #7  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.654] #8  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.654] Thread 6 (Thread 0x7fc837fff700 (LWP 23632) "log-wrlsn-serve"):
       [2025/02/20 17:42:41.654] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.654] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x10994a8, cond=cond@entry=0x1159810, usecs=1000000, run_func=<optimized out>, signalled=signalled@entry=0x7fc837ffdaff) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.654] #2  0x00007fc84d95e025 in __wt_cond_auto_wait_signal (session=session@entry=0x10994a8, cond=0x1159810, progress=<optimized out>, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fc837ffdaff) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:70
       [2025/02/20 17:42:41.654] #3  0x00007fc84d95e185 in __wt_cond_auto_wait (session=session@entry=0x10994a8, cond=<optimized out>, progress=<optimized out>, run_func=run_func@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/cond_auto.c:89
       [2025/02/20 17:42:41.654] #4  0x00007fc84d8b691d in __log_wrlsn_server (arg=0x10994a8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_mgr.c:878
       [2025/02/20 17:42:41.654] #5  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.654] #6  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.654] Thread 5 (Thread 0x7fc83cdf5700 (LWP 23631) "log-close-serve"):
       [2025/02/20 17:42:41.654] #0  0x00007fc84d6a27d1 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.654] #1  0x00007fc84d8d307c in __wt_cond_wait_signal (session=session@entry=0x1098e20, cond=0xc22260, usecs=usecs@entry=100000, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fc83cdf36e0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:115
       [2025/02/20 17:42:41.654] #2  0x00007fc84d8b4473 in __wt_cond_wait (run_func=0x0, usecs=100000, cond=<optimized out>, session=0x1098e20) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/misc_inline.h:21
       [2025/02/20 17:42:41.654] #3  __log_file_server (arg=0x1098e20) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_mgr.c:686
       [2025/02/20 17:42:41.654] #4  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.654] #5  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.654] Thread 4 (Thread 0x7fc83effd700 (LWP 23628) "live_restore 10"):
       [2025/02/20 17:42:41.654] #0  0x00007fc84d43a1eb in select () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.654] #1  0x00007fc84d8d37f3 in __wt_sleep (seconds=seconds@entry=0, micro_seconds=micro_seconds@entry=1000) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_sleep.c:30
       [2025/02/20 17:42:41.654] #2  0x00007fc84d8b921c in __wti_log_slot_join (session=session@entry=0x1097a88, mysize=mysize@entry=128, flags=flags@entry=0, myslot=myslot@entry=0x7fc83eff9630) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_slot.c:606
       [2025/02/20 17:42:41.654] #3  0x00007fc84d8a50d1 in __log_write_internal (session=session@entry=0x1097a88, record=record@entry=0x7fc810004470, lsnp=lsnp@entry=0x0, flags=flags@entry=0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2680
       [2025/02/20 17:42:41.654] #4  0x00007fc84d8a580b in __wt_log_write (session=session@entry=0x1097a88, record=<optimized out>, lsnp=lsnp@entry=0x0, flags=flags@entry=0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2584
       [2025/02/20 17:42:41.693] #5  0x00007fc84d8a5e25 in __wt_log_vprintf (session=0x1097a88, fmt=0x7fc84d9ce510 "RENAME: posix_directory_sync %s", ap=ap@entry=0x7fc83eff9bb8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2794
       [2025/02/20 17:42:41.693] #6  0x00007fc84d8a5efa in __wt_log_printf (session=session@entry=0x1097a88, format=format@entry=0x7fc84d9ce510 "RENAME: posix_directory_sync %s") at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:35
       [2025/02/20 17:42:41.693] #7  0x00007fc84d8d0e43 in __posix_fs_rename (file_system=<optimized out>, wt_session=0x1097a88, from=0x7fc8108b9180 "WT_TEST/WiredTiger.turtle.set", to=0x7fc810768ff0 "WT_TEST/WiredTiger.turtle", flags=1) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_fs.c:358
       [2025/02/20 17:42:41.693] #8  0x00007fc84d898b6e in __live_restore_fs_rename (fs=0xcee200, wt_session=0x1097a88, from=0x7fc8121c66a0 "WT_TEST/WiredTiger.turtle.set", to=0x7fc810066b60 "WT_TEST/WiredTiger.turtle", flags=1) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_fs.c:1700
       [2025/02/20 17:42:41.693] #9  0x00007fc84d8c36d4 in __wt_fs_rename (session=session@entry=0x1097a88, from=from@entry=0x7fc84d9cd582 "WiredTiger.turtle.set", to=to@entry=0x7fc84d9bc490 "WiredTiger.turtle", durable=durable@entry=true) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/os_fs_inline.h:195
       [2025/02/20 17:42:41.693] #10 0x00007fc84d8c606e in __wt_sync_and_rename (to=0x7fc84d9bc490 "WiredTiger.turtle", from=0x7fc84d9cd582 "WiredTiger.turtle.set", fstrp=0x7fc83effa5a8, session=0x1097a88) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/os_fstream_inline.h:93
       [2025/02/20 17:42:41.694] #11 __wti_turtle_update (session=session@entry=0x1097a88, key=key@entry=0x7fc84d9a6a64 "file:WiredTiger.wt", value=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/meta/meta_turtle.c:738
       [2025/02/20 17:42:41.694] #12 0x00007fc84d8c0544 in __wt_metadata_turtle_rewrite (session=session@entry=0x1097a88) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/meta/meta_table.c:56
       [2025/02/20 17:42:41.694] #13 0x00007fc84d89c09d in __wti_live_restore_set_state (session=session@entry=0x1097a88, lr_fs=lr_fs@entry=0xcee200, new_state=new_state@entry=WTI_LIVE_RESTORE_STATE_CLEAN_UP) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_state.c:198
       [2025/02/20 17:42:41.694] #14 0x00007fc84d89a24d in __live_restore_clean_up (session=session@entry=0x1097a88, ctx=ctx@entry=0x4744f80) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_server.c:60
       [2025/02/20 17:42:41.694] #15 0x00007fc84d89a4e3 in __live_restore_worker_stop (session=0x1097a88, ctx=0x4744f80) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_server.c:107
       [2025/02/20 17:42:41.694] #16 0x00007fc84d97af98 in __thread_run (arg=0x4744f80) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/support/thread_group.c:39
       [2025/02/20 17:42:41.694] #17 0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.694] #18 0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.694] Thread 3 (Thread 0x7fc846209700 (LWP 23606) "tiered-server"):
       [2025/02/20 17:42:41.694] #0  0x00007fc84d6a2376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.694] #1  0x00007fc84d8d3385 in __wt_cond_wait_signal (session=session@entry=0x1093938, cond=0x3983830, usecs=usecs@entry=0, run_func=run_func@entry=0x7fc84d809ccb <__tiered_server_run_chk>, signalled=signalled@entry=0x7fc846207aaf) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_posix/os_mtx_cond.c:117
       [2025/02/20 17:42:41.694] #2  0x00007fc84d80baec in __tiered_server (arg=0x1093938) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/conn/conn_tiered.c:450
       [2025/02/20 17:42:41.694] #3  0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.694] #4  0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.694] Thread 2 (Thread 0x7fc845a08700 (LWP 23605) "statlog-server"):
       [2025/02/20 17:42:41.694] #0  0x00007fc84d6a6170 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.694] #1  0x00007fc84d69e0a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.694] #2  0x00007fc84d89c676 in __wt_spin_lock (t=0xcee288, session=0x10932b0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/mutex_inline.h:192
       [2025/02/20 17:42:41.772] #3  __wti_live_restore_get_state (session=session@entry=0x10932b0, lr_fs=0xcee200) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_state.c:269
       [2025/02/20 17:42:41.772] #4  0x00007fc84d89c7c7 in __wti_live_restore_migration_complete (session=session@entry=0x10932b0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_state.c:20
       [2025/02/20 17:42:41.772] #5  0x00007fc84d897944 in __live_restore_setup_lr_fh_file (session=session@entry=0x10932b0, lr_fs=lr_fs@entry=0xcee200, name=name@entry=0x7fc828000c10 "WT_TEST/./WiredTigerStat.20.23", file_type=file_type@entry=WT_FS_OPEN_FILE_TYPE_REGULAR, flags=flags@entry=36, lr_fh=0x7fc828001580) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_fs.c:1518
       [2025/02/20 17:42:41.772] #6  0x00007fc84d89911a in __live_restore_fs_open_file (fs=0xcee200, wt_session=0x10932b0, name=0x7fc828000c10 "WT_TEST/./WiredTigerStat.20.23", file_type=WT_FS_OPEN_FILE_TYPE_REGULAR, flags=36, file_handlep=0x7fc828001568) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_fs.c:1597
       [2025/02/20 17:42:41.772] #7  0x00007fc84d8cb677 in __wt_open (session=session@entry=0x10932b0, name=0x7fc828000c10 "WT_TEST/./WiredTigerStat.20.23", file_type=file_type@entry=WT_FS_OPEN_FILE_TYPE_REGULAR, flags=flags@entry=36, fhp=fhp@entry=0x7fc845a069d8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_common/os_fhandle.c:280
       [2025/02/20 17:42:41.772] #8  0x00007fc84d8cea16 in __wt_fopen (session=session@entry=0x10932b0, name=<optimized out>, open_flags=open_flags@entry=36, flags=flags@entry=1, fstrp=fstrp@entry=0x4a6a458) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/os_common/os_fstream.c:183
       [2025/02/20 17:42:41.772] #9  0x00007fc84d807d3a in __statlog_log_one (session=session@entry=0x10932b0, path=path@entry=0x7fc845a06af0, tmp=tmp@entry=0x7fc845a06ac0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/conn/conn_stat.c:430
       [2025/02/20 17:42:41.772] #10 0x00007fc84d808121 in __statlog_server (arg=0x10932b0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/conn/conn_stat.c:537
       [2025/02/20 17:42:41.773] #11 0x00007fc84d69b609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.773] #12 0x00007fc84d444353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
       [2025/02/20 17:42:41.773] Thread 1 (Thread 0x7fc84d322b00 (LWP 23575) "test_live_resto"):
       [2025/02/20 17:42:41.773] #0  0x00007fc84d6a6170 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.773] #1  0x00007fc84d69e0a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
       [2025/02/20 17:42:41.773] #2  0x00007fc84d897d55 in __wt_spin_lock (t=0xcee288, session=0x109e990) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/mutex_inline.h:192
       [2025/02/20 17:42:41.773] #3  __live_restore_fs_directory_list_worker (fs=0xcee200, wt_session=0x109e990, directory=0x560c9e0 "WT_TEST/journal", prefix=0x7fc84d9c9795 "WiredTigerPreplog", dirlistp=0x7ffd64be06b0, countp=0x7ffd64be06b8, single=true) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_fs.c:255
       [2025/02/20 17:42:41.773] #4  0x00007fc84d8986eb in __live_restore_fs_directory_list_single (fs=<optimized out>, wt_session=<optimized out>, directory=<optimized out>, prefix=<optimized out>, dirlistp=<optimized out>, countp=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/live_restore/live_restore_fs.c:387
       [2025/02/20 17:42:41.773] #5  0x00007fc84d89d70a in __wt_fs_directory_list_single (countp=0x7ffd64be06b8, dirlistp=0x7ffd64be06b0, prefix=0x7fc84d9c9795 "WiredTigerPreplog", dir=<optimized out>, session=0x109e990) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/include/os_fs_inline.h:73
       [2025/02/20 17:42:41.773] #6  __log_get_files_single (session=session@entry=0x109e990, file_prefix=file_prefix@entry=0x7fc84d9c9795 "WiredTigerPreplog", filesp=filesp@entry=0x7ffd64be06b0, countp=countp@entry=0x7ffd64be06b8) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:100
       [2025/02/20 17:42:41.773] #7  0x00007fc84d8a09c4 in __log_alloc_prealloc (session=session@entry=0x109e990, to_num=118) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:1093
       [2025/02/20 17:42:41.773] #8  0x00007fc84d8a1642 in __log_newfile (session=session@entry=0x109e990, conn_open=conn_open@entry=false, created=created@entry=0x7ffd64be0f9f) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:1184
       [2025/02/20 17:42:41.773] #9  0x00007fc84d8a176a in __wti_log_acquire (session=session@entry=0x109e990, recsize=<optimized out>, slot=slot@entry=0x5609990) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:1365
       [2025/02/20 17:42:41.773] #10 0x00007fc84d8b8122 in __log_slot_new (session=session@entry=0x109e990) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_slot.c:273
       [2025/02/20 17:42:41.773] #11 0x00007fc84d8b9753 in __log_slot_switch_internal (session=session@entry=0x109e990, myslot=myslot@entry=0x7ffd64be2110, forced=forced@entry=false, did_work=did_work@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_slot.c:370
       [2025/02/20 17:42:41.773] #12 0x00007fc84d8b9949 in __wti_log_slot_switch (session=session@entry=0x109e990, myslot=myslot@entry=0x7ffd64be2110, retry=retry@entry=true, forced=forced@entry=false, did_work=did_work@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log_slot.c:411
       [2025/02/20 17:42:41.773] #13 0x00007fc84d8a5124 in __log_write_internal (session=session@entry=0x109e990, record=record@entry=0x577a010, lsnp=lsnp@entry=0x0, flags=flags@entry=0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2689
       [2025/02/20 17:42:41.773] #14 0x00007fc84d8a580b in __wt_log_write (session=session@entry=0x109e990, record=<optimized out>, lsnp=lsnp@entry=0x0, flags=0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/log/log.c:2584
       [2025/02/20 17:42:41.773] #15 0x00007fc84d991a15 in __wti_txn_log_commit (session=session@entry=0x109e990, cfg=cfg@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/txn/txn_log.c:321
       [2025/02/20 17:42:41.773] #16 0x00007fc84d98cd78 in __wt_txn_commit (session=session@entry=0x109e990, cfg=cfg@entry=0x0) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/txn/txn.c:1758
       [2025/02/20 17:42:41.773] #17 0x00007fc84d829340 in __curfile_insert (cursor=0xb493610) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/src/cursor/cur_file.c:409
       [2025/02/20 17:42:41.773] #18 0x0000000000406a57 in insert (cursor=..., coll=...) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/test/cppsuite/tests/test_live_restore.cpp:189
       [2025/02/20 17:42:41.773] #19 0x0000000000407828 in write (session=..., fresh_start=fresh_start@entry=true) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/test/cppsuite/tests/test_live_restore.cpp:233
       [2025/02/20 17:42:41.773] #20 0x00000000004079f5 in do_random_crud (session=..., collection_count=collection_count@entry=9223372036854775807, op_count=op_count@entry=200000, fresh_start=fresh_start@entry=false, conn_config=..., home=...) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/test/cppsuite/tests/test_live_restore.cpp:284
       [2025/02/20 17:42:41.773] #21 0x0000000000408930 in run_restore (home=..., source=..., thread_count=thread_count@entry=12, collection_count=collection_count@entry=9223372036854775807, op_count=op_count@entry=200000, background_thread_mode=background_thread_mode@entry=false, verbose_level=0, die=false, recovery=false) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/test/cppsuite/tests/test_live_restore.cpp:367
       [2025/02/20 17:42:41.774] #22 0x00000000004096b6 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/3015b1c167fba15419427c31cce40ed2/wiredtiger/test/cppsuite/tests/test_live_restore.cpp:543
       [2025/02/20 17:42:41.777] warning: target file /proc/23575/cmdline contained unexpected null characters 

            Assignee:
            sean.watt@mongodb.com Sean Watt
            Reporter:
            luke.pearson@mongodb.com Luke Pearson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: