-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Labels:
-
8
-
Storage Engines 2019-12-02, Storage Engines 2019-12-16
This is another cache stuck clean case after the recent merge of WT-5082, on ocelot-aws. In this specific case, the cache was full, no timestamp involved, and there's a core dumped.
http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/25332/
++ nice ./t -1 -c ../../../test/format/CONFIG.stress file_type=var leaf_page_max=9 internal_page_max=9 value_min=256 t: process 31902 running [1575053832:616423][31902:0x7f503f6f9700], t, eviction-server: __evict_server, 440: Cache stuck for too long, giving up: Connection timed out 0x7f5040037e80:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x7f5040037e80:transaction state dump 0x7f5040037e80:current ID: 3 0x7f5040037e80:last running ID: 3 0x7f5040037e80:metadata_pinned ID: 3 0x7f5040037e80:oldest ID: 3 0x7f5040037e80:durable timestamp: (0,0) 0x7f5040037e80:oldest timestamp: (0,0) 0x7f5040037e80:pinned timestamp: (0,0) 0x7f5040037e80:stable timestamp: (0,0) 0x7f5040037e80:has_durable_timestamp: no 0x7f5040037e80:has_oldest_timestamp: no 0x7f5040037e80:has_pinned_timestamp: no 0x7f5040037e80:has_stable_timestamp: no 0x7f5040037e80:oldest_is_pinned: no 0x7f5040037e80:stable_is_pinned: no 0x7f5040037e80:checkpoint running: no 0x7f5040037e80:checkpoint generation: 1 0x7f5040037e80:checkpoint pinned ID: 0 0x7f5040037e80:checkpoint txn ID: 0 0x7f5040037e80:oldest named snapshot ID: 0 0x7f5040037e80:session count: 11 0x7f5040037e80:Transaction state of active sessions: 0x7f5040037e80:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x7f5040037e80:cache dump 0x7f5040037e80:cache full: yes 0x7f5040037e80:cache clean check: yes (115.378%) 0x7f5040037e80:cache dirty check: no (0.003%) 0x7f5040037e80:file:wt.wt(<live>) eviction disabled at open: 0x7f5040037e80: Opened exclusively. Cannot walk tree, skipping. 0x7f5040037e80:file:WiredTigerLAS.wt(<live>) eviction disabled at open: 0x7f5040037e80:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7f5040037e80:leaf: 0 pages 0x7f5040037e80:file:WiredTiger.wt(<live>): 0x7f5040037e80:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7f5040037e80:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7f5040037e80:cache dump: total found: 0MB vs tracked inuse 166MB 0x7f5040037e80:total dirty bytes: 0MB [1575053832:617658][31902:0x7f503f6f9700], t, eviction-server: __wt_evict_thread_run, 315: cache eviction thread error: Connection timed out [1575053832:617743][31902:0x7f503f6f9700], t, eviction-server: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic [1575053832:617819][31902:0x7f503f6f9700], t, eviction-server: __wt_abort, 28: aborting WiredTiger library /tmp/jenkins18201607046064919540.sh: line 28: 31902 Aborted (core dumped) nice ./t -1 -c ../../../test/format/CONFIG.stress file_type=var leaf_page_max=9 internal_page_max=9 value_min=256
The configuration:
############################################ # RUN PARAMETERS ############################################ abort=0 alter=0 assert_commit_timestamp=0 assert_read_timestamp=0 auto_throttle=1 backups=0 bitcnt=1 bloom=1 bloom_bit_count=54 bloom_hash_count=16 bloom_oldest=0 cache=144 cache_minimum=20 checkpoints=on checkpoint_log_size=24 checkpoint_wait=80 checksum=uncompressed chunk_size=5 compaction=0 compression=zlib data_extend=0 data_source=table delete_pct=1 dictionary=0 direct_io=0 encryption=none evict_max=1 file_type=variable-length column-store firstfit=0 huffman_key=0 huffman_value=0 independent_thread_rng=0 in_memory=0 insert_pct=5 internal_key_truncation=1 internal_page_max=9 isolation=snapshot key_gap=11 key_max=78 key_min=28 leaf_page_max=9 leak_memory=0 logging=0 logging_archive=1 logging_compression=none logging_file_max=342005 logging_prealloc=0 long_running_txn=0 lsm_worker_threads=3 memory_page_max=9 merge_max=17 mmap=1 modify_pct=92 ops=0 prefix_compression=1 prefix_compression_min=3 prepare=0 quiet=1 random_cursor=0 read_pct=1 rebalance=1 repeat_data_pct=7 reverse=0 rows=1000000 runs=1 salvage=1 split_pct=63 statistics=0 statistics_server=0 threads=8 timer=4 timing_stress_aggressive_sweep=0 timing_stress_checkpoint=0 timing_stress_lookaside_sweep=0 timing_stress_split_1=0 timing_stress_split_2=0 timing_stress_split_3=0 timing_stress_split_4=0 timing_stress_split_5=0 timing_stress_split_6=0 timing_stress_split_7=0 timing_stress_split_8=0 transaction_timestamps=1 transaction-frequency=100 truncate=1 value_max=827 value_min=256 verify=1 wiredtiger_config= write_pct=1 ############################################