-
Type:
Build Failure
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Test Format
format-stress-test-no-barrier on amazon2023-stress-tests-armv9
Host: i-04cb0c1215e17b2d2
Project: wiredtiger
Commit: f53c3259
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-no-barrier task_log
Logs:
[1765883056:114326][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
format-stress-test-no-barrier task_log
Logs:
0x7203ffe946e8:transaction state dump
0x7203ffe946e8:current ID: 2779
0x7203ffe946e8:last running ID: 2779
0x7203ffe946e8:metadata_pinned ID: 2778
0x7203ffe946e8:oldest ID: 2779
0x7203ffe946e8:durable timestamp: (0, 10376)
0x7203ffe946e8:oldest timestamp: (0, 8040)
0x7203ffe946e8:pinned timestamp: (0, 8040)
0x7203ffe946e8:stable timestamp: (0, 8040)
0x7203ffe946e8:has_durable_timestamp: yes
0x7203ffe946e8:has_oldest_timestamp: yes
0x7203ffe946e8:has_pinned_timestamp: yes
0x7203ffe946e8:has_stable_timestamp: yes
0x7203ffe946e8:oldest_is_pinned: yes
0x7203ffe946e8:stable_is_pinned: yes
0x7203ffe946e8:checkpoint running: yes
0x7203ffe946e8:checkpoint generation: 14
0x7203ffe946e8:checkpoint pinned ID: 2778
0x7203ffe946e8:checkpoint txn ID: 2778
0x7203ffe946e8:session count: 42
0x7203ffe946e8:Transaction state of active sessions:
0x7203ffe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x7203ffe946e8:cache dump
0x7203ffe946e8:cache full: no
0x7203ffe946e8:cache clean check: no (33.147%)
0x7203ffe946e8:cache dirty check: yes (20.136%)
0x7203ffe946e8:cache updates check: no (1.358%)
0x7203ffe946e8:file:F00002(<live>):
0x7203ffe946e8:internal: 37 pages, 50638.67 KB, 37/0 clean/dirty pages, 50638.67/0.00 clean / dirty KB, 1486.54 KB max page, 0.00 KB max dirty page
0x7203ffe946e8:leaf: 11085 pages, 23505.40 KB, 9605/1480 clean/dirty pages, 19591.31 /3914.09 /1272.00 clean/dirty/updates KB, 82.80 KB max page, 82.75 KB max dirty page
0x7203ffe946e8:file:T00001.wt(<live>):
0x7203ffe946e8:internal: 1 pages, 533.62 KB, 1/0 clean/dirty pages, 533.62/0.00 clean / dirty KB, 533.62 KB max page, 0.00 KB max dirty page
0x7203ffe946e8:leaf: 2780 pages, 274940.42 KB, 1834/946 clean/dirty pages, 174397.43 /100542.99 /11394.22 clean/dirty/updates KB, 197.68 KB max page, 197.68 KB max dirty page
0x7203ffe946e8:file:T00003.wt(<live>):
0x7203ffe946e8:internal: 9 pages, 220.18 KB, 2/7 clean/dirty pages, 56.55/163.63 clean / dirty KB, 28.74 KB max page, 27.81 KB max dirty page
0x7203ffe946e8:leaf: 1083 pages, 565329.84 KB, 227/856 clean/dirty pages, 113841.15 /451488.69 /24807.35 clean/dirty/updates KB, 634.37 KB max page, 634.37 KB max dirty page
0x7203ffe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
0x7203ffe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
0x7203ffe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
0x7203ffe946e8:file:WiredTiger.wt(<live>):
0x7203ffe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
0x7203ffe946e8:leaf: 1 pages, 24.21 KB, 0/1 clean/dirty pages, 0.00 /24.21 /17.13 clean/dirty/updates KB, 24.21 KB max page, 24.21 KB max dirty page
0x7203ffe946e8:cache dump: total found: 965.24 MB vs tracked inuse 893.74 MB
0x7203ffe946e8:total dirty bytes: 543.10 MB vs tracked dirty 543.10 MB
format-stress-test-no-barrier task_log
Logs:
[1765883056:120409][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 797: Error at src/btree/bt_curnext.c:797: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1765883056:120419][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1765883056:120422][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
[1765883056:120427][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
[1765883056:120429][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1765883056:120433][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-no-barrier task_log
Logs:
/data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff9a2cea84]
format-stress-test-no-barrier task_log
Logs:
test/format run configuration highlights
format-stress-test-no-barrier task_log
Logs:
#0 0x0000ffff99ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff99ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff99e763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff99e62264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff9a22cf9c [PAC] in __wt_abort (session=session@entry=0x7203ffe946e8) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffff9a2cea84 in __wt_panic_func (session=session@entry=0x7203ffe946e8, error=error@entry=110, func=func@entry=0xffff9a3c1140 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff9a361338 "eviction thread error") at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/support/err.c:611 #5 0x0000ffff9a1e3190 in __evict_thread_run (session=0x7203ffe946e8, thread=0x7203ffc00a50) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/evict/evict_lru.c:359 #6 0x0000ffff9a2e9764 in __thread_run (arg=0x7203ffc00a50) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/support/thread_group.c:32 #7 0x0000ffff99ebdb78 in start_thread () from /lib64/libc.so.6 #8 0x0000ffff99f2acdc [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts: