-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker, Test Format
-
Storage Engines, Storage Engines - Transactions
-
SE Transactions - 2025-11-21
-
5
-
9
format-stress-test-disagg-leader-palite-1 on amazon2023-disagg-stress
Host: i-0a203a27c6aae8131
Project: wiredtiger
Commit: 9a827073
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-leader-palite-1 task_log
Logs:
[1759969804:494917][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 541: Cache stuck for too long, giving up: Connection timed out
format-stress-test-disagg-leader-palite-1 task_log
Logs:
0x5107ffc8a8b0:transaction state dump 0x5107ffc8a8b0:current ID: 710677 0x5107ffc8a8b0:last running ID: 710677 0x5107ffc8a8b0:metadata_pinned ID: 38050 0x5107ffc8a8b0:oldest ID: 710677 0x5107ffc8a8b0:durable timestamp: (0, 647843) 0x5107ffc8a8b0:oldest timestamp: (0, 647794) 0x5107ffc8a8b0:pinned timestamp: (0, 647794) 0x5107ffc8a8b0:stable timestamp: (0, 647794) 0x5107ffc8a8b0:has_durable_timestamp: yes 0x5107ffc8a8b0:has_oldest_timestamp: yes 0x5107ffc8a8b0:has_pinned_timestamp: yes 0x5107ffc8a8b0:has_stable_timestamp: yes 0x5107ffc8a8b0:oldest_is_pinned: yes 0x5107ffc8a8b0:stable_is_pinned: yes 0x5107ffc8a8b0:checkpoint running: yes 0x5107ffc8a8b0:checkpoint generation: 4 0x5107ffc8a8b0:checkpoint pinned ID: 37847 0x5107ffc8a8b0:checkpoint txn ID: 38050 0x5107ffc8a8b0:session count: 22 0x5107ffc8a8b0:Transaction state of active sessions: 0x5107ffc8a8b0:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x5107ffc8a8b0:cache dump 0x5107ffc8a8b0:cache full: no 0x5107ffc8a8b0:cache clean check: no (16.687%) 0x5107ffc8a8b0:cache dirty check: no (15.229%) 0x5107ffc8a8b0:cache updates check: yes (10.000%) 0x5107ffc8a8b0:file:WiredTigerShared.wt_stable(<live>): 0x5107ffc8a8b0:internal: 1 pages, 0.88 KB, 1/0 clean/dirty pages, 0.88/0.00 clean / dirty KB, 0.88 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 0 pages 0x5107ffc8a8b0:file:T00003.wt_stable(<live>): 0x5107ffc8a8b0:internal: 764 pages, 16728.82 KB, 760/4 clean/dirty pages, 15747.04/981.78 clean / dirty KB, 986.29 KB max page, 790.87 KB max dirty page 0x5107ffc8a8b0:leaf: 88422 pages, 351786.79 KB, 9222/79200 clean/dirty pages, 31086.88 /320699.91 /227744.69 clean/dirty/updates KB, 11322.63 KB max page, 11322.63 KB max dirty page 0x5107ffc8a8b0:file:T00003.wt_ingest(<live>) eviction disabled at open: 0x5107ffc8a8b0:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 1 pages, 0.32 KB, 1/0 clean/dirty pages, 0.32 /0.00 /0.00 clean/dirty/updates KB, 0.32 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:file:T00002.wt_stable(<live>): 0x5107ffc8a8b0:internal: 1 pages, 2871.79 KB, 0/1 clean/dirty pages, 0.00/2871.79 clean / dirty KB, 2871.79 KB max page, 2871.79 KB max dirty page 0x5107ffc8a8b0:leaf: 1322 pages, 169184.07 KB, 18/1304 clean/dirty pages, 3023.81 /166160.27 /81908.78 clean/dirty/updates KB, 445.70 KB max page, 445.70 KB max dirty page 0x5107ffc8a8b0:file:T00002.wt_ingest(<live>) eviction disabled at open: 0x5107ffc8a8b0:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 1 pages, 0.32 KB, 1/0 clean/dirty pages, 0.32 /0.00 /0.00 clean/dirty/updates KB, 0.32 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:file:T00001.wt_stable(<live>): 0x5107ffc8a8b0:internal: 1 pages, 1198.43 KB, 0/1 clean/dirty pages, 0.00/1198.43 clean / dirty KB, 1198.43 KB max page, 1198.43 KB max dirty page 0x5107ffc8a8b0:leaf: 5510 pages, 103752.01 KB, 10/5500 clean/dirty pages, 1552.25 /102199.77 /77356.71 clean/dirty/updates KB, 971.64 KB max page, 971.64 KB max dirty page 0x5107ffc8a8b0:file:T00001.wt_ingest(<live>) eviction disabled at open: 0x5107ffc8a8b0:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 1 pages, 0.32 KB, 1/0 clean/dirty pages, 0.32 /0.00 /0.00 clean/dirty/updates KB, 0.32 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:file:WiredTigerSharedHS.wt_stable(<live>) eviction disabled at open: 0x5107ffc8a8b0:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 0 pages 0x5107ffc8a8b0:file:WiredTigerHS.wt(<live>) eviction disabled at open: 0x5107ffc8a8b0:internal: 1 pages, 0.40 KB, 1/0 clean/dirty pages, 0.40/0.00 clean / dirty KB, 0.40 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 0 pages 0x5107ffc8a8b0:file:WiredTiger.wt(<live>): 0x5107ffc8a8b0:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:leaf: 1 pages, 15.00 KB, 1/0 clean/dirty pages, 15.00 /0.00 /0.00 clean/dirty/updates KB, 15.00 KB max page, 0.00 KB max dirty page 0x5107ffc8a8b0:cache dump: total found: 680.85 MB vs tracked inuse 631.34 MB 0x5107ffc8a8b0:total dirty bytes: 580.19 MB vs tracked dirty 581.11 MB
format-stress-test-disagg-leader-palite-1 task_log
Logs:
[1759969804:518170][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 955: Error at src/btree/bt_curnext.c:955: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1759969804:518178][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found [1759969804:518180][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 335: Error at src/evict/evict_lru.c:335: "ret" failed: Connection timed out [1759969804:518183][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 358: eviction thread error: Connection timed out [1759969804:518185][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 358: the process must exit and restart: WT_PANIC: WiredTiger library panic [1759969804:518189][5323:0xffff9c1edbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
format-stress-test-disagg-leader-palite-1 task_log
Logs:
#0 0x0000ffff9d4bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff9d4bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff9d4763a0 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff9d462264 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff9d824d18 [PAC] in __wt_abort (session=session@entry=0x5107ffc8a8b0) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffff9d8c8644 in __wt_panic_func (session=session@entry=0x5107ffc8a8b0, error=error@entry=110, func=func@entry=0xffff9d9b53f0 <__PRETTY_FUNCTION__.29> "__evict_thread_run", line=line@entry=358, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff9d956c78 "eviction thread error") at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/support/err.c:611 #5 0x0000ffff9d7db934 in __evict_thread_run (session=0x5107ffc8a8b0, thread=0x5107ffe090e0) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/evict/evict_lru.c:358 #6 0x0000ffff9d8e2c94 in __thread_run (arg=0x5107ffe090e0) at /data/mci/c2aa74c33623fc2689fb505e9eca986b/wiredtiger/src/support/thread_group.c:32 #7 0x0000ffff9d4bdb78 in start_thread () from /lib64/libc.so.6 #8 0x0000ffff9d52acdc [PAC] in thread_start () from /lib64/libc.so.6
Repro Artifacts:
- is related to
-
WT-15717 Investigate high cache size usage with precise checkpoints enabled in test/format
-
- Open
-