Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Transactions
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-armv9-nonstandalone
- amazon2023-stress-tests-armv9
Failing Tasks:
- format-stress-test-1
- format-stress-test-no-barrier
First Failing Revision:
- 777a8f2d2b927b72ffa0c99a339a780f6d715429(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16295
Count of Linked BFGs (Last 30 days):
3

format-stress-test-no-barrier on amazon2023-armv9-nonstandalone

Host: i-0af47498aa1a63108
Project: wiredtiger
Commit: 777a8f2d
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-no-barrier

format-stress-test-no-barrier task_log

Logs:

    [1765842077:236856][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

format-stress-test-no-barrier task_log

Logs:

    0x12b5bfeb06e8:transaction state dump
    0x12b5bfeb06e8:current ID: 203379
    0x12b5bfeb06e8:last running ID: 203379
    0x12b5bfeb06e8:metadata_pinned ID: 203379
    0x12b5bfeb06e8:oldest ID: 203379
    0x12b5bfeb06e8:durable timestamp: (0, 412938)
    0x12b5bfeb06e8:oldest timestamp: (0, 410604)
    0x12b5bfeb06e8:pinned timestamp: (0, 410604)
    0x12b5bfeb06e8:stable timestamp: (0, 410604)
    0x12b5bfeb06e8:has_durable_timestamp: yes
    0x12b5bfeb06e8:has_oldest_timestamp: yes
    0x12b5bfeb06e8:has_pinned_timestamp: yes
    0x12b5bfeb06e8:has_stable_timestamp: yes
    0x12b5bfeb06e8:oldest_is_pinned: yes
    0x12b5bfeb06e8:stable_is_pinned: yes
    0x12b5bfeb06e8:checkpoint running: no
    0x12b5bfeb06e8:checkpoint generation: 13
    0x12b5bfeb06e8:checkpoint pinned ID: 0
    0x12b5bfeb06e8:checkpoint txn ID: 0
    0x12b5bfeb06e8:session count: 32
    0x12b5bfeb06e8:Transaction state of active sessions:
    0x12b5bfeb06e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    0x12b5bfeb06e8:cache dump
    0x12b5bfeb06e8:cache full: no
    0x12b5bfeb06e8:cache clean check: no (55.536%)
    0x12b5bfeb06e8:cache dirty check: yes (20.020%)
    0x12b5bfeb06e8:cache updates check: no (1.947%)
    0x12b5bfeb06e8:file:T00003.wt(<live>):
    0x12b5bfeb06e8:internal: 463 pages, 161003.20 KB, 233/230 clean/dirty pages, 73246.25/87756.95 clean / dirty KB, 482.61 KB max page, 426.39 KB max dirty page
    0x12b5bfeb06e8:leaf: 14544 pages, 16248.91 KB, 13599/945 clean/dirty pages, 14727.59 /1521.32 /513.89 clean/dirty/updates KB, 81.10 KB max page, 81.10 KB max dirty page
    0x12b5bfeb06e8:file:T00002.wt(<live>):
    0x12b5bfeb06e8:internal: 8926 pages, 34193.68 KB, 8887/39 clean/dirty pages, 34031.42/162.27 clean / dirty KB, 6.70 KB max page, 4.50 KB max dirty page
    0x12b5bfeb06e8:leaf: 13258 pages, 32852.94 KB, 13218/40 clean/dirty pages, 32724.25 /128.69 /47.26 clean/dirty/updates KB, 82.54 KB max page, 4.21 KB max dirty page
    0x12b5bfeb06e8:file:T00001.wt(<live>):
    0x12b5bfeb06e8:internal: 615 pages, 14152.20 KB, 289/326 clean/dirty pages, 6478.30/7673.90 clean / dirty KB, 171.02 KB max page, 171.02 KB max dirty page
    0x12b5bfeb06e8:leaf: 10491 pages, 55415.05 KB, 9570/921 clean/dirty pages, 49160.24 /6254.81 /1431.49 clean/dirty/updates KB, 245.02 KB max page, 245.02 KB max dirty page
    0x12b5bfeb06e8:file:T00005.wt(<live>):
    0x12b5bfeb06e8:internal: 1 pages, 167.92 KB, 0/1 clean/dirty pages, 0.00/167.92 clean / dirty KB, 167.92 KB max page, 167.92 KB max dirty page
    0x12b5bfeb06e8:leaf: 809 pages, 416086.08 KB, 228/581 clean/dirty pages, 118623.66 /297462.42 /27206.37 clean/dirty/updates KB, 644.93 KB max page, 644.93 KB max dirty page
    0x12b5bfeb06e8:file:T00004.wt(<live>):
    0x12b5bfeb06e8:internal: 4 pages, 805.53 KB, 1/3 clean/dirty pages, 1.19/804.34 clean / dirty KB, 382.11 KB max page, 382.11 KB max dirty page
    0x12b5bfeb06e8:leaf: 3237 pages, 381152.19 KB, 2501/736 clean/dirty pages, 285618.37 /95533.83 /9766.97 clean/dirty/updates KB, 212.97 KB max page, 212.97 KB max dirty page
    0x12b5bfeb06e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
    0x12b5bfeb06e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
    0x12b5bfeb06e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
    0x12b5bfeb06e8:file:WiredTiger.wt(<live>):
    0x12b5bfeb06e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
    0x12b5bfeb06e8:leaf: 1 pages, 28.26 KB, 1/0 clean/dirty pages, 28.26 /0.00 /17.04 clean/dirty/updates KB, 28.26 KB max page, 0.00 KB max dirty page
    0x12b5bfeb06e8:cache dump: total found: 1172.93 MB vs tracked inuse 1086.04 MB
    0x12b5bfeb06e8:total dirty bytes: 485.81 MB vs tracked dirty 488.52 MB

format-stress-test-no-barrier task_log

Logs:

    [1765842077:595541][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 797: Error at src/btree/bt_curnext.c:797: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
    [1765842077:595555][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
    [1765842077:595558][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
    [1765842077:595562][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
    [1765842077:595564][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
    [1765842077:595568][16969:0xffffa6fbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

format-stress-test-no-barrier task_log

Logs:

    /data/mci/f8a4afeb34309a8f5432dfe6e9aa907f/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffffaa0cea40]

format-stress-test-no-barrier task_log

Logs:

test/format run configuration highlights

format-stress-test-no-barrier task_log

Logs:

#0  0x0000ffffa9cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffa9cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffa9c763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffa9c62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffaa02cf58 [PAC] in __wt_abort (session=session@entry=0x12b5bfeb06e8) at /data/mci/f8a4afeb34309a8f5432dfe6e9aa907f/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffaa0cea40 in __wt_panic_func (session=session@entry=0x12b5bfeb06e8, error=error@entry=110, func=func@entry=0xffffaa1c1240 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffaa1613a0 "eviction thread error") at /data/mci/f8a4afeb34309a8f5432dfe6e9aa907f/wiredtiger/src/support/err.c:611
#5  0x0000ffffa9fe3188 in __evict_thread_run (session=0x12b5bfeb06e8, thread=0x12b5bfc00140) at /data/mci/f8a4afeb34309a8f5432dfe6e9aa907f/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffffaa0e9718 in __thread_run (arg=0x12b5bfc00140) at /data/mci/f8a4afeb34309a8f5432dfe6e9aa907f/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffffa9cbdb78 in start_thread () from /lib64/libc.so.6
#8  0x0000ffffa9d2acdc [PAC] in thread_start () from /lib64/libc.so.6

Repro Artifacts:

CONFIG

is related to

WT-16299 DUPLICATE cache stuck, bt_curnext not found

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Dec 16 2025 10:23:00 PM UTC
Updated:: Dec 16 2025 11:20:11 PM UTC