Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Transactions
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
- format-stress-test-1
- format-stress-test-2
First Failing Revision:
- 5fff5b92e28e133dac89a560ba439e81ba23b14a(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16237
Count of Linked BFGs (Last 30 days):
4

format-stress-test-1 on amazon2023-armv9-nonstandalone

Host: i-0f6d760ce56739a32
Project: wiredtiger
Commit: 5fff5b92
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-1

format-stress-test-1 task_log

Logs:

    [1765334822:821817][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

format-stress-test-1 task_log

Logs:

    0x5107ffe946e8:transaction state dump
    0x5107ffe946e8:current ID: 38189
    0x5107ffe946e8:last running ID: 38189
    0x5107ffe946e8:metadata_pinned ID: 38189
    0x5107ffe946e8:oldest ID: 38189
    0x5107ffe946e8:durable timestamp: (0, 78022)
    0x5107ffe946e8:oldest timestamp: (0, 76242)
    0x5107ffe946e8:pinned timestamp: (0, 76242)
    0x5107ffe946e8:stable timestamp: (0, 76242)
    0x5107ffe946e8:has_durable_timestamp: yes
    0x5107ffe946e8:has_oldest_timestamp: yes
    0x5107ffe946e8:has_pinned_timestamp: yes
    0x5107ffe946e8:has_stable_timestamp: yes
    0x5107ffe946e8:oldest_is_pinned: yes
    0x5107ffe946e8:stable_is_pinned: yes
    0x5107ffe946e8:checkpoint running: no
    0x5107ffe946e8:checkpoint generation: 12
    0x5107ffe946e8:checkpoint pinned ID: 0
    0x5107ffe946e8:checkpoint txn ID: 0
    0x5107ffe946e8:session count: 40
    0x5107ffe946e8:Transaction state of active sessions:
    0x5107ffe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    0x5107ffe946e8:cache dump
    0x5107ffe946e8:cache full: no
    0x5107ffe946e8:cache clean check: no (23.719%)
    0x5107ffe946e8:cache dirty check: yes (20.086%)
    0x5107ffe946e8:cache updates check: no (1.486%)
    0x5107ffe946e8:file:F00003(<live>):
    0x5107ffe946e8:internal: 31 pages, 16380.20 KB, 9/22 clean/dirty pages, 4265.06/12115.14 clean / dirty KB, 719.83 KB max page, 719.83 KB max dirty page
    0x5107ffe946e8:leaf: 7116 pages, 11309.00 KB, 6641/475 clean/dirty pages, 10262.04 /1046.96 /307.29 clean/dirty/updates KB, 82.13 KB max page, 82.13 KB max dirty page
    0x5107ffe946e8:file:T00002.wt(<live>):
    0x5107ffe946e8:internal: 1 pages, 125.54 KB, 0/1 clean/dirty pages, 0.00/125.54 clean / dirty KB, 125.54 KB max page, 125.54 KB max dirty page
    0x5107ffe946e8:leaf: 553 pages, 239118.94 KB, 61/492 clean/dirty pages, 22433.82 /216685.12 /12375.55 clean/dirty/updates KB, 556.53 KB max page, 556.53 KB max dirty page
    0x5107ffe946e8:file:T00001.wt(<live>):
    0x5107ffe946e8:internal: 13 pages, 81.45 KB, 1/12 clean/dirty pages, 2.73/78.72 clean / dirty KB, 6.83 KB max page, 6.83 KB max dirty page
    0x5107ffe946e8:leaf: 368 pages, 180711.78 KB, 47/321 clean/dirty pages, 21233.51 /159478.27 /15162.20 clean/dirty/updates KB, 600.22 KB max page, 600.22 KB max dirty page
    0x5107ffe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
    0x5107ffe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
    0x5107ffe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
    0x5107ffe946e8:file:WiredTiger.wt(<live>):
    0x5107ffe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
    0x5107ffe946e8:leaf: 1 pages, 17.99 KB, 1/0 clean/dirty pages, 17.99 /0.00 /11.01 clean/dirty/updates KB, 17.99 KB max page, 0.00 KB max dirty page
    0x5107ffe946e8:cache dump: total found: 472.23 MB vs tracked inuse 449.78 MB
    0x5107ffe946e8:total dirty bytes: 380.40 MB vs tracked dirty 392.93 MB

format-stress-test-1 task_log

Logs:

    [1765334822:846795][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846800][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846802][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846803][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846805][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846809][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846810][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846812][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846814][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846815][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846817][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846819][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846820][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846822][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846824][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846825][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846827][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846829][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846830][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846834][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846835][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846837][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846839][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846840][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846842][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846844][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846845][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846847][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846849][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846850][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846852][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846854][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846855][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846857][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846860][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3011: Error at src/evict/evict_lru.c:3011: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765334822:846862][78605:0xffff8d1edbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VE

format-stress-test-1 task_log

Logs:

    /data/mci/5c9d2fce7746e775474ad95e0af92a28/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff8e4cd590]

format-stress-test-1 task_log

Logs:

test/format run configuration highlights

format-stress-test-1 task_log

Logs:

#0  0x0000ffff8e0bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff8e0bf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff8e0763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff8e062264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8e42bd80 [PAC] in __wt_abort (session=session@entry=0x5107ffe946e8) at /data/mci/5c9d2fce7746e775474ad95e0af92a28/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff8e4cd590 in __wt_panic_func (session=session@entry=0x5107ffe946e8, error=error@entry=110, func=func@entry=0xffff8e5be950 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff8e55ed20 "eviction thread error") at /data/mci/5c9d2fce7746e775474ad95e0af92a28/wiredtiger/src/support/err.c:611
#5  0x0000ffff8e3e1fb8 in __evict_thread_run (session=0x5107ffe946e8, thread=0x5107ffc00820) at /data/mci/5c9d2fce7746e775474ad95e0af92a28/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffff8e4e81ac in __thread_run (arg=0x5107ffc00820) at /data/mci/5c9d2fce7746e775474ad95e0af92a28/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffff8e0bdb78 in start_thread () from /lib64/libc.so.6
#8  0x0000ffff8e12acdc [PAC] in thread_start () from /lib64/libc.so.6

Repro Artifacts:

CONFIG

related to

WT-16196 failed: format-stress-test-2 on amazon2023-stress-tests-armv9 [wiredtiger @ 6004b445]

Backlog

WT-16275 Cache stuck, __evict_get_ref failed: WT_NOTFOUND: item not found

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Dec 11 2025 12:43:38 AM UTC
Updated:: Dec 16 2025 11:21:35 PM UTC