Loading...

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Test Format
Labels:
- BB-Tools
- format-cache-stuck

Assigned Teams:

Storage Engines - Transactions
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-stress-nonstandalone
Failing Tasks:
- format-stress-test-2
First Failing Revision:
- dd8d31d6bd3d3bfddbcbf741c560a1248e6330f1(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16294
Count of Linked BFGs (Last 30 days):
1

format-stress-test-2 on amazon2023-stress-nonstandalone

Host: i-064a5da87b8a718f1
Project: wiredtiger
Commit: dd8d31d6
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-2

format-stress-test-2 task_log

Logs:

    [1765844989:792962][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

logs

format-stress-test-2 task_log

Logs:

    0x51397fe9e6e8:transaction state dump
    0x51397fe9e6e8:current ID: 2252
    0x51397fe9e6e8:last running ID: 2252
    0x51397fe9e6e8:metadata_pinned ID: 2252
    0x51397fe9e6e8:oldest ID: 2252
    0x51397fe9e6e8:durable timestamp: (0, 10826)
    0x51397fe9e6e8:oldest timestamp: (0, 9000)
    0x51397fe9e6e8:pinned timestamp: (0, 9000)
    0x51397fe9e6e8:stable timestamp: (0, 9000)
    0x51397fe9e6e8:has_durable_timestamp: yes
    0x51397fe9e6e8:has_oldest_timestamp: yes
    0x51397fe9e6e8:has_pinned_timestamp: yes
    0x51397fe9e6e8:has_stable_timestamp: yes
    0x51397fe9e6e8:oldest_is_pinned: yes
    0x51397fe9e6e8:stable_is_pinned: yes
    0x51397fe9e6e8:checkpoint running: no
    0x51397fe9e6e8:checkpoint generation: 15
    0x51397fe9e6e8:checkpoint pinned ID: 0
    0x51397fe9e6e8:checkpoint txn ID: 0
    0x51397fe9e6e8:session count: 33
    0x51397fe9e6e8:Transaction state of active sessions:
    0x51397fe9e6e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    0x51397fe9e6e8:cache dump
    0x51397fe9e6e8:cache full: no
    0x51397fe9e6e8:cache clean check: no (38.690%)
    0x51397fe9e6e8:cache dirty check: yes (20.126%)
    0x51397fe9e6e8:cache updates check: no (1.468%)
    0x51397fe9e6e8:file:F00003(<live>):
    0x51397fe9e6e8:internal: 20 pages, 230.34 KB, 1/19 clean/dirty pages, 3.99/226.35 clean / dirty KB, 13.93 KB max page, 13.93 KB max dirty page
    0x51397fe9e6e8:leaf: 1114 pages, 554886.13 KB, 369/745 clean/dirty pages, 175695.81 /379190.32 /26699.44 clean/dirty/updates KB, 629.98 KB max page, 614.93 KB max dirty page
    0x51397fe9e6e8:file:T00002.wt(<live>):
    0x51397fe9e6e8:internal: 3269 pages, 64014.82 KB, 2422/847 clean/dirty pages, 47258.26/16756.56 clean / dirty KB, 29.48 KB max page, 21.36 KB max dirty page
    0x51397fe9e6e8:leaf: 16811 pages, 35928.25 KB, 15653/1158 clean/dirty pages, 33023.96 /2904.28 /609.29 clean/dirty/updates KB, 82.01 KB max page, 82.01 KB max dirty page
    0x51397fe9e6e8:file:T00001.wt(<live>):
    0x51397fe9e6e8:internal: 3392 pages, 65347.91 KB, 2543/849 clean/dirty pages, 49123.33/16224.58 clean / dirty KB, 163.07 KB max page, 21.05 KB max dirty page
    0x51397fe9e6e8:leaf: 13532 pages, 24749.68 KB, 12328/1204 clean/dirty pages, 22106.60 /2643.08 /791.76 clean/dirty/updates KB, 82.06 KB max page, 81.82 KB max dirty page
    0x51397fe9e6e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
    0x51397fe9e6e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
    0x51397fe9e6e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
    0x51397fe9e6e8:file:WiredTiger.wt(<live>):
    0x51397fe9e6e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
    0x51397fe9e6e8:leaf: 1 pages, 24.27 KB, 1/0 clean/dirty pages, 24.27 /0.00 /17.20 clean/dirty/updates KB, 24.27 KB max page, 0.00 KB max dirty page
    0x51397fe9e6e8:cache dump: total found: 785.93 MB vs tracked inuse 733.65 MB
    0x51397fe9e6e8:total dirty bytes: 408.15 MB vs tracked dirty 414.08 MB

logs

format-stress-test-2 task_log

Logs:

    [1765844990:131775][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131784][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131787][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131788][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131790][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131792][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131794][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131795][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131797][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131799][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131800][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131802][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131804][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131805][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131809][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131811][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131812][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131814][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131816][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131818][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131819][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131821][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131823][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131824][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131826][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131828][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131829][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131831][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131833][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131836][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131838][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131840][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131841][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131843][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131845][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3025: Error at src/evict/evict_lru.c:3025: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
    [1765844990:131846][48496:0xffff7fdedbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VE

logs

format-stress-test-2 task_log

Logs:

    /data/mci/f90030624db1d8924a9cfca7b12d47c1/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff810ce9ec]

logs

format-stress-test-2 task_log

Logs:

test/format run configuration highlights

logs

format-stress-test-2 task_log

Logs:

#0  0x0000ffff80cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff80cbf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff80c763a0 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff80c62264 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8102cf04 [PAC] in __wt_abort (session=session@entry=0x51397fe9e6e8) at /data/mci/f90030624db1d8924a9cfca7b12d47c1/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff810ce9ec in __wt_panic_func (session=session@entry=0x51397fe9e6e8, error=error@entry=110, func=func@entry=0xffff811c11c0 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff81161320 "eviction thread error") at /data/mci/f90030624db1d8924a9cfca7b12d47c1/wiredtiger/src/support/err.c:611
#5  0x0000ffff80fe3134 in __evict_thread_run (session=0x51397fe9e6e8, thread=0x51397fc00910) at /data/mci/f90030624db1d8924a9cfca7b12d47c1/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffff810e96c4 in __thread_run (arg=0x51397fc00910) at /data/mci/f90030624db1d8924a9cfca7b12d47c1/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffff80cbdb78 in start_thread () from /lib64/libc.so.6
#8  0x0000ffff80d2acdc [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

is related to

WT-16275 Cache stuck, __evict_get_ref failed: WT_NOTFOUND: item not found

Closed

WT-16299 DUPLICATE cache stuck, bt_curnext not found

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates