-
Type:
Build Failure
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Cache and Eviction
-
Storage Engines - Transactions
-
None
-
None
-
1
format-stress-test-2 on amazon2023-stress-tests-armv9
Host: i-00914d9d4d77c2eec
Project: wiredtiger
Commit: 6004b445
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-2 task_log
Logs:
echo "Clone failed after $max_attempts attempts."
format-stress-test-2 task_log
Logs:
[1764988339:486437][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
format-stress-test-2 task_log
Logs:
0x3066bfe946e8:transaction state dump
0x3066bfe946e8:current ID: 1192
0x3066bfe946e8:last running ID: 1192
0x3066bfe946e8:metadata_pinned ID: 1191
0x3066bfe946e8:oldest ID: 1192
0x3066bfe946e8:durable timestamp: (0, 6092)
0x3066bfe946e8:oldest timestamp: (0, 5160)
0x3066bfe946e8:pinned timestamp: (0, 5160)
0x3066bfe946e8:stable timestamp: (0, 5160)
0x3066bfe946e8:has_durable_timestamp: yes
0x3066bfe946e8:has_oldest_timestamp: yes
0x3066bfe946e8:has_pinned_timestamp: yes
0x3066bfe946e8:has_stable_timestamp: yes
0x3066bfe946e8:oldest_is_pinned: yes
0x3066bfe946e8:stable_is_pinned: yes
0x3066bfe946e8:checkpoint running: yes
0x3066bfe946e8:checkpoint generation: 35
0x3066bfe946e8:checkpoint pinned ID: 1191
0x3066bfe946e8:checkpoint txn ID: 1191
0x3066bfe946e8:session count: 48
0x3066bfe946e8:Transaction state of active sessions:
0x3066bfe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x3066bfe946e8:cache dump
0x3066bfe946e8:cache full: no
0x3066bfe946e8:cache clean check: no (29.845%)
0x3066bfe946e8:cache dirty check: yes (20.477%)
0x3066bfe946e8:cache updates check: no (1.689%)
0x3066bfe946e8:file:F00002(<live>):
0x3066bfe946e8:internal: 5 pages, 97.38 KB, 5/0 clean/dirty pages, 97.38/0.00 clean / dirty KB, 28.43 KB max page, 0.00 KB max dirty page
0x3066bfe946e8:leaf: 506 pages, 260019.11 KB, 141/365 clean/dirty pages, 69663.63 /190355.48 /13002.43 clean/dirty/updates KB, 622.00 KB max page, 622.00 KB max dirty page
0x3066bfe946e8:file:T00001.wt(<live>):
0x3066bfe946e8:internal: 137 pages, 941.04 KB, 137/0 clean/dirty pages, 941.04/0.00 clean / dirty KB, 8.16 KB max page, 0.00 KB max dirty page
0x3066bfe946e8:leaf: 2646 pages, 117373.86 KB, 2053/593 clean/dirty pages, 88686.39 /28687.47 /2981.47 clean/dirty/updates KB, 226.43 KB max page, 226.43 KB max dirty page
0x3066bfe946e8:file:T00003.wt(<live>):
0x3066bfe946e8:internal: 1 pages, 66.95 KB, 0/1 clean/dirty pages, 0.00/66.95 clean / dirty KB, 66.95 KB max page, 66.95 KB max dirty page
0x3066bfe946e8:leaf: 350 pages, 201023.22 KB, 42/308 clean/dirty pages, 22462.84 /178560.38 /16805.46 clean/dirty/updates KB, 684.48 KB max page, 684.48 KB max dirty page
0x3066bfe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
0x3066bfe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
0x3066bfe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
0x3066bfe946e8:file:WiredTiger.wt(<live>):
0x3066bfe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
0x3066bfe946e8:leaf: 1 pages, 17.93 KB, 0/1 clean/dirty pages, 0.00 /17.93 /10.94 clean/dirty/updates KB, 17.93 KB max page, 17.93 KB max dirty page
0x3066bfe946e8:cache dump: total found: 611.23 MB vs tracked inuse 565.96 MB
0x3066bfe946e8:total dirty bytes: 388.37 MB vs tracked dirty 388.37 MB
format-stress-test-2 task_log
Logs:
[1764988339:487772][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 797: Error at src/btree/bt_curnext.c:797: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1764988339:487777][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1764988339:487780][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
[1764988339:487783][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
[1764988339:487785][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1764988339:487788][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-2 task_log
Logs:
/data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff882cd1d4]
format-stress-test-2 task_log
Logs:
test/format run configuration highlights
format-stress-test-2 task_log
Logs:
#0 0x0000ffff87ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff87ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff87e763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff87e62264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff8822b9c4 [PAC] in __wt_abort (session=session@entry=0x3066bfe946e8) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffff882cd1d4 in __wt_panic_func (session=session@entry=0x3066bfe946e8, error=error@entry=110, func=func@entry=0xffff883be318 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff8835e840 "eviction thread error") at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/support/err.c:611 #5 0x0000ffff881e1bf8 in __evict_thread_run (session=0x3066bfe946e8, thread=0x3066bfc00870) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/evict/evict_lru.c:359 #6 0x0000ffff882e7ddc in __thread_run (arg=0x3066bfc00870) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/support/thread_group.c:32 #7 0x0000ffff87ebdb78 in start_thread () from /lib64/libc.so.6 #8 0x0000ffff87f2acdc [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts: