Loading...

Type: Build Failure
Resolution: Fixed
Priority: Major - P3
Fix Version/s: WT12.0.0
Affects Version/s: None
Component/s: Cache and Eviction
Labels:
- BB-Tools
- format-cache-stuck

Assigned Teams:

Storage Engines - Transactions
Sprint:
SE Transactions - 2025-12-19
Story Points:
3
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-armv9-nonstandalone
Failing Tasks:
- format-stress-test-3
First Failing Revision:
- 33bdbaa939b5a3bf3c51eaa47999abf5a8eea96c(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16275
Count of Linked BFGs (Last 30 days):
1

format-stress-test-3 on amazon2023-armv9-nonstandalone

Host: i-0d08f8983f301146a
Project: wiredtiger
Commit: 33bdbaa9
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-3

format-stress-test-3 task_log

Logs:

    [1765535181:660744][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

logs

format-stress-test-3 task_log

Logs:

    0x52ad7fe946e8:transaction state dump
    0x52ad7fe946e8:current ID: 2240
    0x52ad7fe946e8:last running ID: 2240
    0x52ad7fe946e8:metadata_pinned ID: 2240
    0x52ad7fe946e8:oldest ID: 2240
    0x52ad7fe946e8:durable timestamp: (0, 9913)
    0x52ad7fe946e8:oldest timestamp: (0, 8040)
    0x52ad7fe946e8:pinned timestamp: (0, 8040)
    0x52ad7fe946e8:stable timestamp: (0, 8040)
    0x52ad7fe946e8:has_durable_timestamp: yes
    0x52ad7fe946e8:has_oldest_timestamp: yes
    0x52ad7fe946e8:has_pinned_timestamp: yes
    0x52ad7fe946e8:has_stable_timestamp: yes
    0x52ad7fe946e8:oldest_is_pinned: yes
    0x52ad7fe946e8:stable_is_pinned: yes
    0x52ad7fe946e8:checkpoint running: no
    0x52ad7fe946e8:checkpoint generation: 16
    0x52ad7fe946e8:checkpoint pinned ID: 0
    0x52ad7fe946e8:checkpoint txn ID: 0
    0x52ad7fe946e8:session count: 38
    0x52ad7fe946e8:Transaction state of active sessions:
    0x52ad7fe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    0x52ad7fe946e8:cache dump
    0x52ad7fe946e8:cache full: no
    0x52ad7fe946e8:cache clean check: no (40.711%)
    0x52ad7fe946e8:cache dirty check: yes (20.063%)
    0x52ad7fe946e8:cache updates check: no (1.764%)
    0x52ad7fe946e8:file:T00002.wt(<live>):
    0x52ad7fe946e8:internal: 1 pages, 316.60 KB, 0/1 clean/dirty pages, 0.00/316.60 clean / dirty KB, 316.60 KB max page, 316.60 KB max dirty page
    0x52ad7fe946e8:leaf: 1641 pages, 422886.84 KB, 792/849 clean/dirty pages, 196626.69 /226260.15 /16801.26 clean/dirty/updates KB, 364.06 KB max page, 364.06 KB max dirty page
    0x52ad7fe946e8:file:F00001(<live>):
    0x52ad7fe946e8:internal: 1768 pages, 35995.36 KB, 1054/714 clean/dirty pages, 21105.43/14889.93 clean / dirty KB, 395.18 KB max page, 395.18 KB max dirty page
    0x52ad7fe946e8:leaf: 12038 pages, 44902.58 KB, 11107/931 clean/dirty pages, 40917.05 /3985.53 /766.73 clean/dirty/updates KB, 85.60 KB max page, 84.35 KB max dirty page
    0x52ad7fe946e8:file:T00003.wt(<live>):
    0x52ad7fe946e8:internal: 21 pages, 333.84 KB, 1/20 clean/dirty pages, 4.18/329.66 clean / dirty KB, 19.43 KB max page, 19.43 KB max dirty page
    0x52ad7fe946e8:leaf: 1625 pages, 280719.48 KB, 775/850 clean/dirty pages, 126732.31 /153987.17 /16221.98 clean/dirty/updates KB, 261.24 KB max page, 261.24 KB max dirty page
    0x52ad7fe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
    0x52ad7fe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
    0x52ad7fe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
    0x52ad7fe946e8:file:WiredTiger.wt(<live>):
    0x52ad7fe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
    0x52ad7fe946e8:leaf: 1 pages, 17.92 KB, 1/0 clean/dirty pages, 17.92 /0.00 /10.95 clean/dirty/updates KB, 17.92 KB max page, 0.00 KB max dirty page
    0x52ad7fe946e8:cache dump: total found: 828.11 MB vs tracked inuse 771.99 MB
    0x52ad7fe946e8:total dirty bytes: 390.40 MB vs tracked dirty 395.62 MB

logs

format-stress-test-3 task_log

Logs:

    [1765535181:709039][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709047][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709050][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709051][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709053][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709057][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709058][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709060][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709062][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709063][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709065][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709067][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709068][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709070][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709072][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709073][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709075][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709077][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709078][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709080][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709083][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709085][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709086][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709088][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709090][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709091][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709093][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709094][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709096][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709098][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709099][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709101][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709103][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709104][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709106][180891:0xffffa9bbdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765535181:709109][180891:0xffffa9bbdbc0], t, file:Wire

logs

format-stress-test-3 task_log

Logs:

    /data/mci/8442fc5ecdafd1f7b81df01dac0a974c/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffffaccce578]

logs

format-stress-test-3 task_log

Logs:

test/format run configuration highlights

logs

format-stress-test-3 task_log

Logs:

#0  0x0000ffffac8bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffac8bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffac8763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffac862264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffacc2cbf4 [PAC] in __wt_abort (session=session@entry=0x52ad7fe946e8) at /data/mci/8442fc5ecdafd1f7b81df01dac0a974c/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffaccce578 in __wt_panic_func (session=session@entry=0x52ad7fe946e8, error=error@entry=110, func=func@entry=0xffffacdbfc58 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffacd60300 "eviction thread error") at /data/mci/8442fc5ecdafd1f7b81df01dac0a974c/wiredtiger/src/support/err.c:611
#5  0x0000ffffacbe2e24 in __evict_thread_run (session=0x52ad7fe946e8, thread=0x52ad7fc00500) at /data/mci/8442fc5ecdafd1f7b81df01dac0a974c/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffffacce91a8 in __thread_run (arg=0x52ad7fc00500) at /data/mci/8442fc5ecdafd1f7b81df01dac0a974c/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffffac8bdb78 in start_thread () from /lib64/libc.so.6
#8  0x0000ffffac92acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is related to

WT-16237 failed: format-stress-test-1 on amazon2023-armv9-nonstandalone [wiredtiger @ 5fff5b92]

Backlog

related to

WT-16294 format-stress-test-2 cache stuck, __evict_get_ref failed: WT_NOTFOUND: item not found

Backlog

Details

Description

Attachments

Issue Links

Activity

People

Dates