-
Type:
Task
-
Resolution: Done
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
test/fops has been failing in Jenkins because it is hanging. I can only reproduce this on tinderbox, not any other machine I've tried. This is not always reproducible. Often running again Jenkins passes.
Eviction is hung, and it is holding the dhandle lock so everything else is blocked.
3 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__session_checkpoint,obj_checkpoint,fop,start_thread,clone
2 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__wt_curfile_open,__wt_open_cursor,__wt_curtable_open,__wt_open_cursor,__session_open_cursor,obj_bulk_unique,fop,start_thread,clone
2
1 pthread_join,fop_start,main
1 pthread_cond_timedwait@@GLIBC_2.3.2,__wt_cond_wait,__sweep_server,start_thread,clone
1 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__wt_session_get_btree,__wt_session_get_btree_ckpt,__wt_curfile_open,__wt_open_cursor,__wt_curtable_open,__wt_open_cursor,__session_open_cursor,obj_cursor,fop,start_thread,clone
1 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__wt_session_get_btree,__wt_session_get_btree_ckpt,__wt_curfile_open,__wt_open_cursor,__wt_curtable_open,__wt_open_cursor,__session_open_cursor,obj_bulk_unique,fop,start_thread,clone
1 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__wt_session_get_btree,__wt_metadata_open,__schema_add_table,__wt_schema_get_table,__create_table,__wt_schema_create,__session_create,obj_bulk,fop,start_thread,clone
1 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__wt_curfile_open,__wt_open_cursor,__wt_curtable_open,__wt_open_cursor,__session_open_cursor,obj_bulk,fop,start_thread,clone
1 __lll_lock_wait,_L_lock_812,__GI___pthread_mutex_lock,__wt_spin_lock,__session_create,obj_bulk,fop,start_thread,clone
1 __evict_walk,__evict_lru_walk,__evict_pass,__evict_server,start_thread,clone
Here's the stack
Thread 13 (Thread 0x7f7e24a3a700 (LWP 26032)): #0 0x00007f7e24ac9a0d in __evict_walk (flags=<optimized out>, session=0x151d7f0) at ../src/evict/evict_lru.c:901 WT-1 __evict_lru_walk (flags=<optimized out>, session=0x151d7f0) at ../src/evict/evict_lru.c:725 WT-2 __evict_pass (session=0x151d7f0) at ../src/evict/evict_lru.c:484 WT-3 __evict_server (arg=0x151d7f0) at ../src/evict/evict_lru.c:163
Putting breakpoints at key lines in *evict_walk shows that there are four open files and every single one of them has WT_BTREE_NO_EVICTION set. Therefore *evict_walk never returns anything to evict.