-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
3
-
Storage Engines 2020-01-27
We have seen test/format failures with:
- Timestamps enabled
- Many threads configured
- A small cache size
- Random isolation level
- No or small number of active transactions
- No active checkpoint
- Many dirty leaf pages
That are failing due to the cache dirty threshold was reached. We should reproduce and understand why those pages aren't being evicted. If possible evict them, if not figure out how to prevent the cache becoming stuck dirty in format for these configurations.
Test failure:
http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress/67978/
++ nice catchsegv ./t -1 -c ../../../test/format/CONFIG.stress [1537145255:78342][24491:0x7fa8843f1700], t, eviction-server: __evict_server, 451: Cache stuck for too long, giving up: Connection timed out t: process 24491 0x7fa886df57b8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x7fa886df57b8:transaction state dump 0x7fa886df57b8:current ID: 4976395 0x7fa886df57b8:last running ID: 4976395 0x7fa886df57b8:oldest ID: 4976395 0x7fa886df57b8:commit timestamp: 1a84fc 0x7fa886df57b8:oldest timestamp: 1a84fc 0x7fa886df57b8:pinned timestamp: 1a84fc 0x7fa886df57b8:stable timestamp: 0 0x7fa886df57b8:has_commit_timestamp: yes 0x7fa886df57b8:has_oldest_timestamp: yes 0x7fa886df57b8:has_pinned_timestamp: yes 0x7fa886df57b8:has_stable_timestamp: no 0x7fa886df57b8:oldest_is_pinned: yes 0x7fa886df57b8:stable_is_pinned: no 0x7fa886df57b8:checkpoint running: no 0x7fa886df57b8:checkpoint generation: 23 0x7fa886df57b8:checkpoint pinned ID: 0 0x7fa886df57b8:checkpoint txn ID: 0 0x7fa886df57b8:oldest named snapshot ID: 0 0x7fa886df57b8:session count: 14 0x7fa886df57b8:Transaction state of active sessions: 0x7fa886df57b8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x7fa886df57b8:cache dump 0x7fa886df57b8:cache full: no 0x7fa886df57b8:cache clean check: no (63.956%) 0x7fa886df57b8:cache dirty check: yes (20.004%) 0x7fa886df57b8:file:wt.wt(<live>): 0x7fa886df57b8:internal: 21 pages, 3MB, 8/13 clean/dirty pages, 0/2 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7fa886df57b8:leaf: 15777 pages, 22MB, 13802/1975 clean/dirty pages, 13/8 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7fa886df57b8:file:WiredTigerLAS.wt(<live>): 0x7fa886df57b8:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7fa886df57b8:leaf: 0 pages 0x7fa886df57b8:file:WiredTiger.wt(<live>): 0x7fa886df57b8:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7fa886df57b8:leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page 0x7fa886df57b8:cache dump: total found: 28MB vs tracked inuse 28MB 0x7fa886df57b8:total dirty bytes: 11MB [1537145255:83231][24491:0x7fa8843f1700], t, eviction-server: __wt_evict_thread_run, 321: cache eviction thread error: Connection timed out [1537145255:83237][24491:0x7fa8843f1700], t, eviction-server: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic [1537145255:83242][24491:0x7fa8843f1700], t, eviction-server: __wt_abort, 30: aborting WiredTiger library *** Aborted
The configuration:
############################################ # RUN PARAMETERS ############################################ abort=0 alter=0 auto_throttle=1 backups=0 bitcnt=3 bloom=1 bloom_bit_count=13 bloom_hash_count=17 bloom_oldest=0 cache=45 cache_minimum=20 checkpoints=on checkpoint_log_size=182 checkpoint_wait=76 checksum=uncompressed chunk_size=8 compaction=1 compression=snappy data_extend=0 data_source=table delete_pct=8 dictionary=0 direct_io=0 encryption=none evict_max=4 file_type=row-store firstfit=0 huffman_key=0 huffman_value=0 independent_thread_rng=0 in_memory=0 insert_pct=78 internal_key_truncation=1 internal_page_max=16 isolation=random key_gap=3 key_max=58 key_min=15 leaf_page_max=11 leak_memory=0 logging=0 logging_archive=0 logging_compression=none logging_file_max=360989 logging_prealloc=0 long_running_txn=0 lsm_worker_threads=3 merge_max=19 mmap=1 modify_pct=14 ops=0 prefix_compression=1 prefix_compression_min=8 prepare=1 quiet=1 read_pct=0 rebalance=1 repeat_data_pct=53 reverse=0 rows=1000000 runs=1 salvage=1 split_pct=77 statistics=0 statistics_server=0 threads=32 timer=4 timing_stress_checkpoint=0 timing_stress_lookaside_sweep=0 timing_stress_split_1=0 timing_stress_split_2=0 timing_stress_split_3=0 timing_stress_split_4=0 timing_stress_split_5=0 timing_stress_split_6=0 timing_stress_split_7=0 timing_stress_split_8=0 transaction_timestamps=1 transaction-frequency=72 truncate=1 value_max=1280 value_min=16 verify=1 wiredtiger_config= write_pct=0 ############################################
Edit, KB:
This configuration is no longer supported, mixing non-snapshot-isolation transactions with timestamps.