Application threads are tasked with eviction even when pinning the oldest transaction ID

XMLWordPrintableJSON

    • Storage Engines
    • Storage Engines 2019-11-18, Storage Engines 2019-12-02
    • 5

      Summary:
      Detect and avoid more cases where an application thread, pinning the oldest transaction ID, can be tasked with eviction.

      There's a hung of the test/format stress job on zSeries machine.

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-zseries/49467/console

      Used gdb generated the core, and here is the stack:

      (gdb) bt
#0  0x000003ffa290c566 in __pthread_cond_timedwait (cond=0x93098200, mutex=0x930981d8, abstime=0x3ffa21fec68)
    at pthread_cond_timedwait.c:166
#1  0x00000000800954e4 in __wt_cond_wait_signal (session=0x3ffa2bcb590, cond=0x930981d0, usecs=10000,
    run_func=0x0, signalled=0x3ffa21fed7f) at ../src/os_posix/os_mtx_cond.c:121
#2  0x00000000800626a8 in __wt_cond_wait (session=0x3ffa2bcb590, cond=0x930981d0, usecs=10000, run_func=0x0)
    at ../src/include/misc.i:19
#3  0x0000000080067c6e in __evict_lru_pages (session=0x3ffa2bcb590, is_server=false)
    at ../src/evict/evict_lru.c:1126
#4  0x0000000080065970 in __wt_evict_thread_run (session=0x3ffa2bcb590, thread=0x930982b0)
    at ../src/evict/evict_lru.c:314
#5  0x00000000800f7bb2 in __thread_run (arg=0x930982b0) at ../src/support/thread_group.c:31
#6  0x000003ffa29081f2 in start_thread (arg=0x3ffa21ff910) at pthread_create.c:310
#7  0x000003ffa26898da in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:76 

      Cache is full of clean page:

      0x3ffa2bcc3a0:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x3ffa2bcc3a0:cache dump
0x3ffa2bcc3a0:cache full: yes
0x3ffa2bcc3a0:cache clean check: yes (114.145%)
0x3ffa2bcc3a0:cache dirty check: no (2.281%)
0x3ffa2bcc3a0:file:wt(<live>):
0x3ffa2bcc3a0:internal: 680 pages, 81MB, 667/13 clean/dirty pages, 76/4 clean/dirty MB, 2MB max page, 2MB max dirty page
0x3ffa2bcc3a0:leaf: 0 pages
0x3ffa2bcc3a0:file:WiredTigerLAS.wt(<live>):
0x3ffa2bcc3a0:internal: 3 pages, 2MB, 1/2 clean/dirty pages, 0/2 clean/dirty MB, 2MB max page, 2MB max dirty page
0x3ffa2bcc3a0:leaf: 2 pages, 1MB, 0/2 clean/dirty pages, 0/1 clean/dirty MB, 1MB max page, 1MB max dirty page
0x3ffa2bcc3a0:file:WiredTiger.wt(<live>):
0x3ffa2bcc3a0:internal: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
0x3ffa2bcc3a0:leaf: 1 pages, 0MB, 1/0 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
0x3ffa2bcc3a0:cache dump: total found: 92MB vs tracked inuse 92MB
0x3ffa2bcc3a0:total dirty bytes: 8MB
[1566954009:134326][25525:0x3ffa09ff910], t, eviction-server: __wt_evict_thread_run, 318: cache eviction thread error: Connection timed out
[1566954009:134351][25525:0x3ffa09ff910], t, eviction-server: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1566954009:134374][25525:0x3ffa09ff910], t, eviction-server: __wt_abort, 28: aborting WiredTiger library
*** Aborted

      The configuration:

      ############################################
#  RUN PARAMETERS
############################################
abort=0
alter=0
assert_commit_timestamp=0
assert_read_timestamp=0
auto_throttle=1
backups=0
bitcnt=6
bloom=1
bloom_bit_count=28
bloom_hash_count=20
bloom_oldest=0
cache=81
cache_minimum=20
checkpoints=wiredtiger
checkpoint_log_size=119
checkpoint_wait=7
checksum=uncompressed
chunk_size=10
compaction=0
compression=snappy
data_extend=0
data_source=file
delete_pct=13
dictionary=0
direct_io=0
encryption=none
evict_max=0
file_type=row-store
firstfit=0
huffman_key=0
huffman_value=0
independent_thread_rng=1
in_memory=0
insert_pct=9
internal_key_truncation=1
internal_page_max=9
isolation=snapshot
key_gap=2
key_max=256
key_min=256
leaf_page_max=9
leak_memory=0
logging=0
logging_archive=1
logging_compression=none
logging_file_max=295668
logging_prealloc=0
long_running_txn=0
lsm_worker_threads=4
memory_page_max=2
merge_max=12
mmap=1
modify_pct=22
ops=0
prefix_compression=1
prefix_compression_min=5
prepare=0
quiet=1
read_pct=50
rebalance=1
repeat_data_pct=85
reverse=1
rows=1000000
runs=1
salvage=1
split_pct=94
statistics=0
statistics_server=0
threads=20
timer=4
timing_stress_aggressive_sweep=0
timing_stress_checkpoint=0
timing_stress_lookaside_sweep=0
timing_stress_split_1=0
timing_stress_split_2=0
timing_stress_split_3=0
timing_stress_split_4=0
timing_stress_split_5=0
timing_stress_split_6=0
timing_stress_split_7=0
timing_stress_split_8=0
transaction_timestamps=1
transaction-frequency=100
truncate=1
value_max=2385
value_min=15
verify=1
wiredtiger_config=
write_pct=6
############################################

            Assignee:
            Keith Bostic (Inactive)
            Reporter:
            Luke Chen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: