Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2608

update-checkpoint-btree hangs in WT-2544

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None

      Currently wiredtiger-perf-checkpoint is hung on the update-checkpoint-btree task.

      Job is here

      Trace is as below:

      (gdb) thread apply all bt
      
      Thread 9 (Thread 0x7f52ce7ff700 (LWP 16225)):
      #0  0x00007f52cf768f4d in __lll_lock_wait () from /lib64/libpthread.so.0
      #1  0x00007f52cf764d02 in _L_lock_791 () from /lib64/libpthread.so.0
      #2  0x00007f52cf764c08 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3  0x000000000041fde4 in __wt_spin_lock (session=0x7f52ce89bd00, t=<optimized out>) at ../src/include/mutex.i:159
      #4  __evict_clear_walk (session=0x7f52ce89bd00, session=0x7f52ce89bd00, is_locked=false) at ../src/evict/evict_lru.c:731
      #5  __evict_clear_walks (session=session@entry=0x7f52ce89bd00) at ../src/evict/evict_lru.c:759
      #6  0x00000000004215f3 in __evict_pass (is_server=true, session=0x7f52ce89bd00) at ../src/evict/evict_lru.c:590
      #7  __evict_server (arg=0x7f52ce89bd00) at ../src/evict/evict_lru.c:198
      #8  0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 8 (Thread 0x7f52cdffe700 (LWP 16226)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89c040, cond=0x7f52ce80f080, usecs=<optimized out>, signalled=signalled@entry=0x7f52cdffde90) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x000000000041844c in __wt_cond_wait (usecs=<optimized out>, cond=<optimized out>, session=0x7f52ce89c040) at ../src/include/misc.i:18
      #3  __sweep_server (arg=0x7f52ce89c040) at ../src/conn/conn_sweep.c:272
      #4  0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 7 (Thread 0x7f52ccffc700 (LWP 16227)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89c380, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52ccffbb6f) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000421f10 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89c380) at ../src/include/misc.i:18
      #3  __evict_request_clear_walk (session=0x7f52ce89c380) at ../src/evict/evict_lru.c:805
      #4  __wt_evict_file_exclusive_on (session=session@entry=0x7f52ce89c380) at ../src/evict/evict_lru.c:858
      #5  0x0000000000491d99 in __sync_file (syncop=WT_SYNC_CHECKPOINT, session=0x7f52ce89c380) at ../src/btree/bt_sync.c:127
      #6  __wt_cache_op (session=session@entry=0x7f52ce89c380, op=op@entry=WT_SYNC_CHECKPOINT) at ../src/btree/bt_sync.c:293
      #7  0x000000000047076a in __checkpoint_tree (cfg=<optimized out>, is_checkpoint=true, session=0x7f52ce89c380) at ../src/txn/txn_ckpt.c:1125
      #8  __checkpoint_tree_helper (session=0x7f52ce89c380, cfg=<optimized out>) at ../src/txn/txn_ckpt.c:1205
      #9  0x0000000000402ae9 in __checkpoint_apply (session=session@entry=0x7f52ce89c380, cfg=cfg@entry=0x7f52ccffbe30, op=op@entry=0x470400 <__checkpoint_tree_helper>) at ../src/txn/txn_ckpt.c:186
      #10 0x0000000000471622 in __txn_checkpoint (session=session@entry=0x7f52ce89c380, cfg=cfg@entry=0x7f52ccffbe30) at ../src/txn/txn_ckpt.c:507
      #11 0x0000000000471e02 in __wt_txn_checkpoint (session=session@entry=0x7f52ce89c380, cfg=cfg@entry=0x7f52ccffbe30) at ../src/txn/txn_ckpt.c:668
      #12 0x000000000045ceae in __session_checkpoint (wt_session=0x7f52ce89c380, config=<optimized out>) at ../src/session/session_api.c:1455
      #13 0x0000000000407776 in checkpoint_worker (arg=0x7f52ce8c0000) at ../../../bench/wtperf/wtperf.c:1402
      #14 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #15 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 6 (Thread 0x7f52c8fff700 (LWP 16228)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89c6c0, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52c8ffe88f) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000422343 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89c6c0) at ../src/include/misc.i:18
      #3  __wt_cache_eviction_worker (session=0x7f52ce89c6c0, busy=<optimized out>, pct_full=<optimized out>) at ../src/evict/evict_lru.c:1815
      #4  0x00000000004ecc63 in __wt_cache_eviction_check (busy=false, didworkp=0x0, session=0x7f52ce89c6c0) at ../src/include/cache.i:226
      #5  __cursor_enter (session=0x7f52ce89c6c0) at ../src/include/cursor.i:59
      #6  __curfile_enter (cbt=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:93
      #7  __cursor_func_init (reenter=true, cbt=0x7f52c9421600) at ../src/include/cursor.i:266
      #8  __wt_btcur_insert (cbt=cbt@entry=0x7f52c9421600) at ../src/btree/bt_cursor.c:520
      #9  0x00000000004b19ae in __curfile_insert (cursor=0x7f52c9421600) at ../src/cursor/cur_file.c:245
      #10 0x0000000000409006 in worker (arg=<optimized out>) at ../../../bench/wtperf/wtperf.c:670
      #11 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #12 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 5 (Thread 0x7f52c87fe700 (LWP 16229)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89d080, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52c87fd8cf) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000422343 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89d080) at ../src/include/misc.i:18
      #3  __wt_cache_eviction_worker (session=0x7f52ce89d080, busy=<optimized out>, pct_full=<optimized out>) at ../src/evict/evict_lru.c:1815
      #4  0x00000000004e97dc in __wt_cache_eviction_check (busy=false, didworkp=0x0, session=0x7f52ce89d080) at ../src/include/cache.i:226
      #5  __cursor_enter (session=0x7f52ce89d080) at ../src/include/cursor.i:59
      #6  __curfile_enter (cbt=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:93
      #7  __cursor_func_init (reenter=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:266
      #8  __wt_btcur_search (cbt=cbt@entry=0x7f52ca021600) at ../src/btree/bt_cursor.c:337
      #9  0x00000000004b0ea3 in __curfile_search (cursor=0x7f52ca021600) at ../src/cursor/cur_file.c:200
      #10 0x0000000000409188 in worker (arg=<optimized out>) at ../../../bench/wtperf/wtperf.c:639
      #11 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #12 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 4 (Thread 0x7f52c7ffd700 (LWP 16230)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89cd40, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52c7ffc8cf) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000422343 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89cd40) at ../src/include/misc.i:18
      #3  __wt_cache_eviction_worker (session=0x7f52ce89cd40, busy=<optimized out>, pct_full=<optimized out>) at ../src/evict/evict_lru.c:1815
      #4  0x00000000004e97dc in __wt_cache_eviction_check (busy=false, didworkp=0x0, session=0x7f52ce89cd40) at ../src/include/cache.i:226
      #5  __cursor_enter (session=0x7f52ce89cd40) at ../src/include/cursor.i:59
      #6  __curfile_enter (cbt=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:93
      #7  __cursor_func_init (reenter=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:266
      #8  __wt_btcur_search (cbt=cbt@entry=0x7f52c9c21600) at ../src/btree/bt_cursor.c:337
      #9  0x00000000004b0ea3 in __curfile_search (cursor=0x7f52c9c21600) at ../src/cursor/cur_file.c:200
      #10 0x0000000000409188 in worker (arg=<optimized out>) at ../../../bench/wtperf/wtperf.c:639
      #11 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      ---Type <return> to continue, or q <return> to quit---
      #12 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 3 (Thread 0x7f52c77fc700 (LWP 16231)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89ca00, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52c77fb88f) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000422343 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89ca00) at ../src/include/misc.i:18
      #3  __wt_cache_eviction_worker (session=0x7f52ce89ca00, busy=<optimized out>, pct_full=<optimized out>) at ../src/evict/evict_lru.c:1815
      #4  0x00000000004f1243 in __wt_cache_eviction_check (busy=false, didworkp=0x0, session=0x7f52ce89ca00) at ../src/include/cache.i:226
      #5  __cursor_enter (session=0x7f52ce89ca00) at ../src/include/cursor.i:59
      #6  __curfile_enter (cbt=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:93
      #7  __cursor_func_init (reenter=true, cbt=0x7f52c9821600) at ../src/include/cursor.i:266
      #8  __wt_btcur_update (cbt=cbt@entry=0x7f52c9821600) at ../src/btree/bt_cursor.c:776
      #9  0x00000000004b15c3 in __curfile_update (cursor=0x7f52c9821600) at ../src/cursor/cur_file.c:284
      #10 0x0000000000409512 in worker (arg=<optimized out>) at ../../../bench/wtperf/wtperf.c:708
      #11 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #12 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 2 (Thread 0x7f52c6ffb700 (LWP 16232)):
      #0  0x00007f52cf766a82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1  0x0000000000435c09 in __wt_cond_wait_signal (session=session@entry=0x7f52ce89d3c0, cond=0x7f52ce80f180, usecs=usecs@entry=100000, signalled=signalled@entry=0x7f52c6ffa8cf) at ../src/os_posix/os_mtx_cond.c:82
      #2  0x0000000000422343 in __wt_cond_wait (usecs=100000, cond=<optimized out>, session=0x7f52ce89d3c0) at ../src/include/misc.i:18
      #3  __wt_cache_eviction_worker (session=0x7f52ce89d3c0, busy=<optimized out>, pct_full=<optimized out>) at ../src/evict/evict_lru.c:1815
      #4  0x00000000004e97dc in __wt_cache_eviction_check (busy=false, didworkp=0x0, session=0x7f52ce89d3c0) at ../src/include/cache.i:226
      #5  __cursor_enter (session=0x7f52ce89d3c0) at ../src/include/cursor.i:59
      #6  __curfile_enter (cbt=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:93
      #7  __cursor_func_init (reenter=<optimized out>, cbt=<optimized out>) at ../src/include/cursor.i:266
      #8  __wt_btcur_search (cbt=cbt@entry=0x7f52ca421600) at ../src/btree/bt_cursor.c:337
      #9  0x00000000004b0ea3 in __curfile_search (cursor=0x7f52ca421600) at ../src/cursor/cur_file.c:200
      #10 0x00000000004090b1 in worker (arg=<optimized out>) at ../../../bench/wtperf/wtperf.c:686
      #11 0x00007f52cf762dc5 in start_thread () from /lib64/libpthread.so.0
      #12 0x00007f52cf48fc9d in clone () from /lib64/libc.so.6
      
      Thread 1 (Thread 0x7f52d04da780 (LWP 16220)):
      #0  0x00007f52cf763ef7 in pthread_join () from /lib64/libpthread.so.0
      #1  0x000000000040b584 in stop_threads (threads=<optimized out>, num=<optimized out>, cfg=0x7ffcb2f83890) at ../../../bench/wtperf/wtperf.c:2490
      #2  execute_workload (cfg=0x7ffcb2f83890) at ../../../bench/wtperf/wtperf.c:1753
      #3  start_run (cfg=cfg@entry=0x7ffcb2f83890) at ../../../bench/wtperf/wtperf.c:2083
      #4  0x000000000040446e in start_all_runs (cfg=0x7ffcb2f83890) at ../../../bench/wtperf/wtperf.c:1931
      #5  main (argc=<optimized out>, argv=<optimized out>) at ../../../bench/wtperf/wtperf.c:2404
      

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            david.hows David Hows
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: