Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools
- CI-blocker

Assigned Teams:

Storage Engines
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- 78adcb70edede9e97b3edf97ad0ea17f95a4e22a(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-15928
Count of Linked BFGs (Last 30 days):
23

format-stress-data-validation-test-disagg-follower-2 on ubuntu2004-arm64-nonstandalone

Host: i-062cde4cdfe05a56a
Project: wiredtiger
Commit: 5269df79
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-data-validation-test-disagg-follower-2

format-stress-data-validation-test-disagg-follower-2 task_log

Logs:

[1761809302:677552][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

logs

format-stress-data-validation-test-disagg-follower-2 task_log

Logs:

0x73663fc881e8:transaction state dump
0x73663fc881e8:current ID: 8098766
0x73663fc881e8:last running ID: 8098766
0x73663fc881e8:metadata_pinned ID: 92
0x73663fc881e8:oldest ID: 92
0x73663fc881e8:durable timestamp: (0, 96857272)
0x73663fc881e8:oldest timestamp: (0, 96857084)
0x73663fc881e8:pinned timestamp: (0, 96857084)
0x73663fc881e8:stable timestamp: (0, 96857084)
0x73663fc881e8:has_durable_timestamp: yes
0x73663fc881e8:has_oldest_timestamp: yes
0x73663fc881e8:has_pinned_timestamp: yes
0x73663fc881e8:has_stable_timestamp: yes
0x73663fc881e8:oldest_is_pinned: yes
0x73663fc881e8:stable_is_pinned: yes
0x73663fc881e8:checkpoint running: no
0x73663fc881e8:checkpoint generation: 74
0x73663fc881e8:checkpoint pinned ID: 0
0x73663fc881e8:checkpoint txn ID: 0
0x73663fc881e8:session count: 38
0x73663fc881e8:Transaction state of active sessions:
0x73663fc881e8:session ID: 34, txn ID: 0, pinned ID: 92, metadata pinned ID: 0, name: WT_CURSOR.next
0x73663fc881e8:transaction id: 0, mod count: 0, snap min: 92, snap max: 142, snapshot count: 11, snapshot: [92, 123, 128, 130, 132, 135, 136, 138, 139, 140, 141], commit_timestamp: (0, 0), durable_timestamp: (0, 0), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), prepared id: 0, pinned_durable_timestamp: (0, 0), read_timestamp: (0, 0), checkpoint LSN: [0,0], full checkpoint: false, flags: 0x00000010, isolation: WT_ISO_SNAPSHOT, last saved error code: 0, last saved sub-level error code: -32000, last saved error message: last API call was successful
0x73663fc881e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x73663fc881e8:cache dump
0x73663fc881e8:cache full: yes
0x73663fc881e8:cache clean check: yes (101.282%)
0x73663fc881e8:cache dirty check: no (76.954%)
0x73663fc881e8:cache updates check: no (80.231%)
0x73663fc881e8:file:T00003.wt_ingest(<live>):
0x73663fc881e8:internal: 1 pages, 113.78 KB, 0/1 clean/dirty pages, 0.00/113.78 clean / dirty KB, 113.78 KB max page, 113.78 KB max dirty page
0x73663fc881e8:leaf: 635 pages, 174995.77 KB, 2/633 clean/dirty pages, 1569.10 /173426.67 /127333.54 clean/dirty/updates KB, 953.65 KB max page, 545.24 KB max dirty page
0x73663fc881e8:file:T00002.wt_ingest(<live>):
0x73663fc881e8:internal: 796 pages, 52378.07 KB, 0/796 clean/dirty pages, 0.00/52378.07 clean / dirty KB, 1647.45 KB max page, 1647.45 KB max dirty page
0x73663fc881e8:leaf: 284818 pages, 1796879.39 KB, 166/284652 clean/dirty pages, 11346.68 /1785532.71 /1415893.24 clean/dirty/updates KB, 2921.67 KB max page, 2921.67 KB max dirty page
0x73663fc881e8:file:T00001.wt_ingest(<live>):
0x73663fc881e8:internal: 1 pages, 2432.50 KB, 0/1 clean/dirty pages, 0.00/2432.50 clean / dirty KB, 2432.50 KB max page, 2432.50 KB max dirty page
0x73663fc881e8:leaf: 4724 pages, 1592531.43 KB, 2/4722 clean/dirty pages, 7286.67 /1585244.76 /1324470.60 clean/dirty/updates KB, 3836.89 KB max page, 2921.67 KB max dirty page
0x73663fc881e8:file:WiredTigerSharedHS.wt_stable(<live>) eviction disabled at open:
0x73663fc881e8:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page
0x73663fc881e8:leaf: 0 pages
0x73663fc881e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
0x73663fc881e8:internal: 1 pages, 0.40 KB, 1/0 clean/dirty pages, 0.40/0.00 clean / dirty KB, 0.40 KB max page, 0.00 KB max dirty page
0x73663fc881e8:leaf: 0 pages
0x73663fc881e8:file:WiredTiger.wt(<live>):
0x73663fc881e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
0x73663fc881e8:leaf: 1 pages, 55.64 KB, 1/0 clean/dirty pages, 55.64 /0.00 /45.48 clean/dirty/updates KB, 55.64 KB max page, 0.00 KB max dirty page
0x73663fc881e8:cache dump: total found: 3817.32 MB vs tracked inuse 3536.61 MB
0x73663fc881e8:total dirty bytes: 3514.77 MB vs tracked dirty 2742.43 MB

logs

format-stress-data-validation-test-disagg-follower-2 task_log

Logs:

[1761809302:829556][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829568][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1761809302:829573][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829575][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829578][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829580][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829582][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829585][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829587][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829589][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829591][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829593][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829596][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829598][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829600][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829604][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829606][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829608][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829610][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829612][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829615][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829617][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829619][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829621][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829624][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829626][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829628][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829630][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829632][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829636][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829638][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829640][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829643][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829645][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 2984: Error at src/evict/evict_lru.c:2984: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1761809302:829647][30145:0xffff9392ac80], t, file:WiredTigerSharedHS.wt_stab

logs

format-stress-data-validation-test-disagg-follower-2 task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffff9503daac in __GI_abort () at abort.c:79
#2  0x0000ffff954f94a4 in __wt_abort (session=session@entry=0x73663fc881e8) at /data/mci/de1f4591ca8194a492bc928158140766/wiredtiger/src/os_common/os_abort.c:32
#3  0x0000ffff9559e174 in __wt_panic_func (session=session@entry=0x73663fc881e8, error=error@entry=110, func=func@entry=0xffff9568d550 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff9562dd58 "eviction thread error") at /data/mci/de1f4591ca8194a492bc928158140766/wiredtiger/src/support/err.c:611
#4  0x0000ffff954afa1c in __evict_thread_run (session=0x73663fc881e8, thread=0x73663fe0b9a0) at /data/mci/de1f4591ca8194a492bc928158140766/wiredtiger/src/evict/evict_lru.c:359
#5  0x0000ffff955b8a48 in __thread_run (arg=0x73663fe0b9a0) at /data/mci/de1f4591ca8194a492bc928158140766/wiredtiger/src/support/thread_group.c:32
#6  0x0000ffff95292624 in start_thread (arg=0xffff955b89f4 <__thread_run>) at pthread_create.c:477
#7  0x0000ffff950ee66c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

logs

Repro Artifacts:

CONFIG

related to

WT-15723 (disagg.mode=leader, PALITE) test/format cache stuck

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Oct 31 2025 06:36:15 AM UTC
Updated:: Nov 03 2025 02:17:20 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates