Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3271

Eviction tuning stuck in a loop

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Minor - P4 Minor - P4
    • WT2.9.2, 3.2.13, 3.4.4, 3.5.6
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Storage 2017-04-17

      test/csuite/test_wt2719_reconfig timed out.

      test_wt2719_reconfig is the test where we throw random configuration and re-configuration stuff at a WiredTiger database, and there's a 60-second timer so we catch hangs and get a core dump.

      My guess is that valgrind slowed things down enough that we crossed the 60-second timer, but we should look through the core.

      If there's no evidence of a real failure, we should increase the timer so valgrind doesn't hit it.

      Here are the stacks:

      Thread 38 (LWP 31760):
      #0  __evict_tune_workers (session=0x5837690) at ../src/evict/evict_lru.c:1001
      #1  0x000000000042308a in __evict_pass (session=0x5837690)
          at ../src/evict/evict_lru.c:660
      #2  0x0000000000422849 in __evict_server (session=0x5837690, 
          did_work=0x641beaf) at ../src/evict/evict_lru.c:408
      #3  0x00000000004224a2 in __wt_evict_thread_run (session=0x5837690, 
          thread=0x58c2310) at ../src/evict/evict_lru.c:313
      #4  0x000000000048a950 in __thread_run (arg=0x58c2310)
          at ../src/support/thread_group.c:31
      #5  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #6  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 37 (LWP 31768):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5837a08, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x6c1ce4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x5837a08, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x5837a08, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x5837a08, 
          thread=0x58c86a0) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0x58c86a0)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 36 (LWP 31777):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5837d80, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x741de4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x5837d80, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x5837d80, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x5837d80, 
          thread=0x58cea30) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0x58cea30)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 35 (LWP 31787):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x58380f8, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x7c1ee4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x58380f8, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x58380f8, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x58380f8, 
          thread=0x58d4dc0) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0x58d4dc0)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 34 (LWP 31798):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x5838470, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x841fe4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x5838470, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #5  0x000000000042430d in __evict_lru_pages (session=0x5838470, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x5838470, 
          thread=0x58db150) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0x58db150)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 33 (LWP 31808):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x58387e8, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x8c20e4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x58387e8, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x58387e8, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x58387e8, 
          thread=0x58e14e0) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0x58e14e0)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 32 (LWP 31819):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x5838b60, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x9421e4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x5838b60, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      ---Type <return> to continue, or q <return> to quit---
      #5  0x000000000042430d in __evict_lru_pages (session=0x5838b60, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x5838b60, 
          thread=0x58e7870) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0x58e7870)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 31 (LWP 32527):
      #0  0x000000000524dc3d in open64 () from /lib64/libpthread.so.0
      #1  0x0000000000447cc2 in __posix_open_file (file_system=0x582ee60, 
          wt_session=0x5838ed8, 
          name=0xfa51410 "WT_TEST.test_wt2719_reconfig/./WiredTigerStat.07.19", 
          file_type=WT_FS_OPEN_FILE_TYPE_REGULAR, flags=68, file_handlep=0xfa68838)
          at ../src/os_posix/os_fs.c:658
      #2  0x000000000044345d in __wt_open (session=0x5838ed8, 
          name=0xfa51410 "WT_TEST.test_wt2719_reconfig/./WiredTigerStat.07.19", 
          file_type=WT_FS_OPEN_FILE_TYPE_REGULAR, flags=68, fhp=0x15439d90)
          at ../src/os_common/os_fhandle.c:258
      #3  0x0000000000445e73 in __wt_fopen (session=0x5838ed8, 
          name=0xfa51410 "WT_TEST.test_wt2719_reconfig/./WiredTigerStat.07.19", 
          open_flags=68, flags=1, fstrp=0x15439de8)
          at ../src/os_common/os_fstream.c:189
      #4  0x0000000000415919 in __statlog_log_one (session=0x5838ed8, 
          path=0x15439ec0, tmp=0x15439e90) at ../src/conn/conn_stat.c:430
      #5  0x0000000000415cc5 in __statlog_server (arg=0x5838ed8)
          at ../src/conn/conn_stat.c:543
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 30 (LWP 31842):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5839250, 
          cond=0x58fa380, usecs=50000000000, 
          run_func=0x416b75 <__sweep_server_run_chk>, signalled=0xa423eaf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000416302 in __wt_cond_wait (session=0x5839250, cond=0x58fa380, 
          usecs=50000000000, run_func=0x416b75 <__sweep_server_run_chk>)
          at ../src/include/misc.i:19
      #3  0x0000000000416bf4 in __sweep_server (arg=0x5839250)
          at ../src/conn/conn_sweep.c:281
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 29 (LWP 31849):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=10000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x00000000004355c2 in __lsm_manager_run_server (session=0x5839940)
          at ../src/lsm/lsm_manager.c:385
      #3  0x00000000004359fa in __lsm_worker_manager (arg=0x5822ca8)
          at ../src/lsm/lsm_manager.c:479
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 28 (LWP 31851):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5839cb8, 
          cond=0x582ea40, usecs=10000, run_func=0x0, signalled=0xb425ebf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000043b9d9 in __wt_cond_wait (session=0x5839cb8, cond=0x582ea40, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000043bfc8 in __lsm_worker (arg=0x5822cd0)
          at ../src/lsm/lsm_worker.c:170
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 27 (LWP 31854):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583a030, 
          cond=0x582ea40, usecs=10000, run_func=0x0, signalled=0xbc26ebf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000043b9d9 in __wt_cond_wait (session=0x583a030, cond=0x582ea40, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000043bfc8 in __lsm_worker (arg=0x5822cf8)
          at ../src/lsm/lsm_worker.c:170
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 26 (LWP 32525):
      #0  0x000000000524d1bd in __lll_lock_wait () from /lib64/libpthread.so.0
      #1  0x0000000005248d02 in _L_lock_791 () from /lib64/libpthread.so.0
      #2  0x0000000005248c08 in pthread_mutex_lock () from /lib64/libpthread.so.0
      #3  0x0000000000420cc5 in __wt_spin_lock (session=0x583dea0, t=0x5856628)
          at ../src/include/mutex.i:169
      #4  0x0000000000420fa7 in __wt_spin_lock_track (session=0x583dea0, t=0x5856628)
          at ../src/include/mutex.i:308
      #5  0x000000000042388f in __wt_evict_file_exclusive_on (session=0x583dea0)
          at ../src/evict/evict_lru.c:868
      #6  0x00000000004ca4d6 in __sync_file (session=0x583dea0, 
          syncop=WT_SYNC_CHECKPOINT) at ../src/btree/bt_sync.c:192
      #7  0x00000000004ca8c3 in __wt_cache_op (session=0x583dea0, 
          op=WT_SYNC_CHECKPOINT) at ../src/btree/bt_sync.c:317
      #8  0x000000000049290a in __checkpoint_tree (session=0x583dea0, 
          is_checkpoint=true, cfg=0x1f84de70) at ../src/txn/txn_ckpt.c:1462
      #9  0x0000000000492d74 in __wt_checkpoint (session=0x583dea0, cfg=0x1f84de70)
          at ../src/txn/txn_ckpt.c:1618
      #10 0x00000000004911e5 in __txn_checkpoint (session=0x583dea0, cfg=0x1f84de70)
          at ../src/txn/txn_ckpt.c:831
      #11 0x0000000000491832 in __txn_checkpoint_wrapper (session=0x583dea0, 
          cfg=0x1f84de70) at ../src/txn/txn_ckpt.c:948
      #12 0x0000000000491995 in __wt_txn_checkpoint (session=0x583dea0, 
          cfg=0x1f84de70, waiting=true) at ../src/txn/txn_ckpt.c:1001
      #13 0x0000000000479236 in __session_checkpoint (wt_session=0x583dea0, 
          config=0x0) at ../src/session/session_api.c:1656
      ---Type <return> to continue, or q <return> to quit---
      #14 0x000000000040f180 in __ckpt_server (arg=0x583dea0)
          at ../src/conn/conn_ckpt.c:116
      #15 0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #16 0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 25 (LWP 32528):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583e218, 
          cond=0xfa5da20, usecs=1000000, run_func=0x0, signalled=0x9c22ecf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000040d10e in __wt_cond_wait (session=0x583e218, cond=0xfa5da20, 
          usecs=1000000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000040edf6 in __wt_cache_pool_server (arg=0x583e218)
          at ../src/conn/conn_cache_pool.c:765
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 24 (LWP 32529):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583e590, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x13435e4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x583e590, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x583e590, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x583e590, 
          thread=0xfa5dee0) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0xfa5dee0)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 23 (LWP 32530):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x583e908, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x13c36e4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x583e908, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #5  0x000000000042430d in __evict_lru_pages (session=0x583e908, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x583e908, 
          thread=0xfa5e830) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0xfa5e830)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 22 (LWP 32532):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583ec80, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0xf02ce4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x583ec80, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x583ec80, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x583ec80, 
          thread=0xfa5f180) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0xfa5f180)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 21 (LWP 32485):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583a3a8, 
          cond=0x582ea40, usecs=10000, run_func=0x0, signalled=0x17c3eebf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000043b9d9 in __wt_cond_wait (session=0x583a3a8, cond=0x582ea40, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000043bfc8 in __lsm_worker (arg=0x5822d20)
          at ../src/lsm/lsm_worker.c:170
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 20 (LWP 32486):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x583a720, 
          cond=0x582ea40, usecs=10000, run_func=0x0, signalled=0x1883febf)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000043b9d9 in __wt_cond_wait (session=0x583a720, cond=0x582ea40, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000043bfc8 in __lsm_worker (arg=0x5822d48)
          at ../src/lsm/lsm_worker.c:170
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 19 (LWP 32488):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x583eff8, op=0x19040eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x583eff8)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 18 (LWP 32489):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      ---Type <return> to continue, or q <return> to quit---
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x583f370, op=0xc827eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x583f370)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 17 (LWP 32490):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x583f6e8, op=0xd028eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x583f6e8)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 16 (LWP 32491):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x583fa60, op=0xd829eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x583fa60)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 15 (LWP 32492):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x583fdd8, op=0xe02aeb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x583fdd8)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 14 (LWP 32493):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x5840150, op=0xe82beb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x5840150)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 13 (LWP 32494):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x58404c8, op=0x1042eeb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x58404c8)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 12 (LWP 32495):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x5840840, op=0x10c2feb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x5840840)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 11 (LWP 32496):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x5840bb8, op=0x12433eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x5840bb8)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 10 (LWP 32497):
      #0  0x0000000005549ba3 in select () from /lib64/libc.so.6
      #1  0x0000000000448efe in __wt_sleep (seconds=0, micro_seconds=100000)
          at ../src/os_posix/os_sleep.c:24
      #2  0x000000000049dd58 in __async_op_dequeue (conn=0x581c200, 
          session=0x5840f30, op=0x12c34eb8) at ../src/async/async_worker.c:52
      #3  0x000000000049e81b in __wt_async_worker (arg=0x5840f30)
          at ../src/async/async_worker.c:301
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 9 (LWP 32533):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x58412a8, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0xf82de4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x58412a8, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #5  0x000000000042430d in __evict_lru_pages (session=0x58412a8, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x58412a8, 
      ---Type <return> to continue, or q <return> to quit---
          thread=0xfa5fd40) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0xfa5fd40)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 8 (LWP 32536):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5841620, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x11430e4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x5841620, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x5841620, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x5841620, 
          thread=0xfa60c60) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0xfa60c60)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 7 (LWP 32537):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x5841998, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x11c31e4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x5841998, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #5  0x000000000042430d in __evict_lru_pages (session=0x5841998, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x5841998, 
          thread=0xfa61820) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0xfa61820)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 6 (LWP 32538):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5841d10, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x14437e4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x5841d10, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x5841d10, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x5841d10, 
          thread=0xfa62450) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0xfa62450)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 5 (LWP 32539):
      #0  0x000000000524d25a in __lll_unlock_wake () from /lib64/libpthread.so.0
      #1  0x0000000005249c53 in _L_unlock_697 () from /lib64/libpthread.so.0
      #2  0x0000000005249bde in pthread_mutex_unlock () from /lib64/libpthread.so.0
      #3  0x0000000000448bb3 in __wt_cond_wait_signal (session=0x5842088, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x14c38e4f)
          at ../src/os_posix/os_mtx_cond.c:107
      #4  0x0000000000421b01 in __wt_cond_wait (session=0x5842088, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #5  0x000000000042430d in __evict_lru_pages (session=0x5842088, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #6  0x00000000004225cc in __wt_evict_thread_run (session=0x5842088, 
          thread=0xfa63c20) at ../src/evict/evict_lru.c:339
      #7  0x000000000048a950 in __thread_run (arg=0xfa63c20)
          at ../src/support/thread_group.c:31
      #8  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #9  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 4 (LWP 32540):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5842400, 
          cond=0x58c21d0, usecs=10000, run_func=0x0, signalled=0x15c3ae4f)
          at ../src/os_posix/os_mtx_cond.c:87
      #2  0x0000000000421b01 in __wt_cond_wait (session=0x5842400, cond=0x58c21d0, 
          usecs=10000, run_func=0x0) at ../src/include/misc.i:19
      #3  0x000000000042430d in __evict_lru_pages (session=0x5842400, 
          is_server=false) at ../src/evict/evict_lru.c:1155
      #4  0x00000000004225cc in __wt_evict_thread_run (session=0x5842400, 
          thread=0xfa65700) at ../src/evict/evict_lru.c:339
      #5  0x000000000048a950 in __thread_run (arg=0xfa65700)
          at ../src/support/thread_group.c:31
      #6  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #7  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 3 (LWP 32541):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5842778, 
          cond=0xfa66b50, usecs=10000000, run_func=0x4223cc <__wt_evict_thread_chk>, 
          signalled=0x1643becf) at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000048a7ec in __wt_cond_wait (session=0x5842778, cond=0xfa66b50, 
          usecs=10000000, run_func=0x4223cc <__wt_evict_thread_chk>)
          at ../src/include/misc.i:19
      #3  0x000000000048a938 in __thread_run (arg=0xfa662c0)
          at ../src/support/thread_group.c:29
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 2 (LWP 32542):
      #0  0x000000000524aa82 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
      ---Type <return> to continue, or q <return> to quit---
         from /lib64/libpthread.so.0
      #1  0x0000000000448b45 in __wt_cond_wait_signal (session=0x5842af0, 
          cond=0xfa67710, usecs=10000000, run_func=0x4223cc <__wt_evict_thread_chk>, 
          signalled=0x16c3cecf) at ../src/os_posix/os_mtx_cond.c:87
      #2  0x000000000048a7ec in __wt_cond_wait (session=0x5842af0, cond=0xfa67710, 
          usecs=10000000, run_func=0x4223cc <__wt_evict_thread_chk>)
          at ../src/include/misc.i:19
      #3  0x000000000048a938 in __thread_run (arg=0xfa66e80)
          at ../src/support/thread_group.c:29
      #4  0x0000000005246dc5 in start_thread () from /lib64/libpthread.so.0
      #5  0x000000000555273d in clone () from /lib64/libc.so.6
      
      Thread 1 (LWP 31724):
      #0  0x00000000054901d7 in raise () from /lib64/libc.so.6
      #1  0x00000000054918c8 in abort () from /lib64/libc.so.6
      #2  0x0000000000402429 in on_alarm (signo=14)
          at ../../../test/csuite/wt2719_reconfig/main.c:203
      #3  <signal handler called>
      #4  0x0000000005247ef5 in pthread_join () from /lib64/libpthread.so.0
      #5  0x0000000000449179 in __wt_thread_join (session=0x58368b0, tid=528803584)
          at ../src/os_posix/os_thread.c:51
      #6  0x000000000040f48c in __wt_checkpoint_server_destroy (session=0x58368b0)
          at ../src/conn/conn_ckpt.c:231
      #7  0x000000000040f3b6 in __wt_checkpoint_server_create (session=0x58368b0, 
          cfg=0xffefffb20) at ../src/conn/conn_ckpt.c:206
      #8  0x0000000000408a73 in __conn_reconfigure (wt_conn=0x581c200, 
          config=0x59b0c90 ",lsm_manager=(worker_thread_max=18),statistics_log=(json=0),file_manager=(close_scan_interval=50000),shared_cache=(reserve=5MB),async=(threads=7),eviction_dirty_target=8,statistics_log=(timestamp=\"%b:"...)
          at ../src/conn/conn_api.c:1140
      #9  0x0000000000402472 in reconfig (opts=0xffefffbf0, session=0x58395c8, 
          config=0x59b0c90 ",lsm_manager=(worker_thread_max=18),statistics_log=(json=0),file_manager=(close_scan_interval=50000),shared_cache=(reserve=5MB),async=(threads=7),eviction_dirty_target=8,statistics_log=(timestamp=\"%b:"...)
          at ../../../test/csuite/wt2719_reconfig/main.c:221
      #10 0x00000000004028c1 in main (argc=1, argv=0xffefffd98)
          at ../../../test/csuite/wt2719_reconfig/main.c:300
      

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: