Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Cache and Eviction
Labels:

Assigned Teams:

Storage Engines - Transactions
Total Hours with Assigned Team:
335.769
Epic Link:
SPM-4852
Sprint:
SE Transactions - 2026-08-14, SE Transactions - 2026-08-28
Story Points:
5
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- bc1fecf502464cee3c89351ffb9b6ee5310dab71(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-18241
Count of Linked BFGs (Last 30 days):
20

format-stress-test-disagg-switch-data-validation-stepdown-async-2 on amazon2023-stress-nonstandalone

Host: i-0cfd1d471f67433f3
Project: wiredtiger
Commit: bc1fecf5
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-data-validation-stepdown-async-2

format-stress-test-disagg-switch-data-validation-stepdown-async-2 task_log

Logs:

[443/688] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o
[447/688] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic

format-stress-test-disagg-switch-data-validation-stepdown-async-2 task_log

Logs:

[1785492042:952745][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 278: Cache stuck for too long, giving up: Connection timed out

format-stress-test-disagg-switch-data-validation-stepdown-async-2 task_log

Logs:

0x10d1bfc918e0:transaction state dump
0x10d1bfc918e0:current ID: 13438478
0x10d1bfc918e0:last running ID: 13438478
0x10d1bfc918e0:metadata_pinned ID: 13438478
0x10d1bfc918e0:oldest ID: 13438478
0x10d1bfc918e0:durable timestamp: (0, 14014578)
0x10d1bfc918e0:oldest timestamp: (0, 11989373)
0x10d1bfc918e0:pinned timestamp: (0, 11989373)
0x10d1bfc918e0:stable timestamp: (0, 12001083)
0x10d1bfc918e0:stable disaggregated schema epoch: (0, 0)
0x10d1bfc918e0:has_durable_timestamp: yes
0x10d1bfc918e0:has_oldest_timestamp: yes
0x10d1bfc918e0:has_pinned_timestamp: yes
0x10d1bfc918e0:has_stable_timestamp: yes
0x10d1bfc918e0:has_stable_disaggregated_schema_epoch: no
0x10d1bfc918e0:oldest_is_pinned: yes
0x10d1bfc918e0:stable_is_pinned: no
0x10d1bfc918e0:checkpoint running: no
0x10d1bfc918e0:checkpoint generation: 23
0x10d1bfc918e0:checkpoint pinned ID: 0
0x10d1bfc918e0:checkpoint txn ID: 0
0x10d1bfc918e0:session count: 19
0x10d1bfc918e0:Transaction state of active sessions:
0x10d1bfc918e0:session ID: 18, txn ID: 0, pinned ID: 13438478, metadata pinned ID: 0, name: WT_CURSOR.next
0x10d1bfc918e0:transaction id: 0, mod count: 0, snap min: 13438478, snap max: 13438478, snapshot count: 0, snapshot: [], commit_timestamp: (0, 0), durable_timestamp: (0, 0), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), prepared id: 0, pinned_durable_timestamp: (0, 0), read_timestamp: (0, 0), checkpoint LSN: [0,0], full checkpoint: false, flags: 0x00000004, isolation: WT_ISO_SNAPSHOT, last saved error code: 0, last saved sub-level error code: -32000, last saved error message: last API call was successful
0x10d1bfc918e0:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x10d1bfc918e0:cache dump
0x10d1bfc918e0:cache full: no
0x10d1bfc918e0:cache clean check: yes (95.001%)
0x10d1bfc918e0:cache dirty check: yes (44.382%)
0x10d1bfc918e0:cache updates check: yes (25.300%)
0x10d1bfc918e0:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.22(<live>):
0x10d1bfc918e0:internal: 1 pages, 1.41 KB, 1/0 clean/dirty pages, 1.41/0.00 clean / dirty KB, 1.41 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 0 pages
0x10d1bfc918e0:file:T00002.wt_stable/WiredTigerCheckpoint.22(<live>):
0x10d1bfc918e0:internal: 251 pages, 4338.84 KB, 251/0 clean/dirty pages, 4338.84/0.00 clean / dirty KB, 162.34 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 16786 pages, 472856.98 KB, 16786/0 clean/dirty pages, 472856.98 /0.00 /0.00 clean/dirty/updates KB, 188.59 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:WiredTigerSharedHS.wt_stable(<live>):
0x10d1bfc918e0:internal: 1 pages, 1.41 KB, 1/0 clean/dirty pages, 1.41/0.00 clean / dirty KB, 1.41 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 0 pages
0x10d1bfc918e0:file:T00002.wt_ingest(<live>):
0x10d1bfc918e0:internal: 1298 pages, 23614.53 KB, 0/1298 clean/dirty pages, 0.00/23614.53 clean / dirty KB, 227.13 KB max page, 227.13 KB max dirty page
0x10d1bfc918e0:leaf: 129741 pages, 324013.22 KB, 0/129741 clean/dirty pages, 0.00 /324013.22 /83993.13 clean/dirty/updates KB, 374.35 KB max page, 374.35 KB max dirty page
0x10d1bfc918e0:file:T00003.wt_ingest(<live>):
0x10d1bfc918e0:internal: 1 pages, 6.22 KB, 0/1 clean/dirty pages, 0.00/6.22 clean / dirty KB, 6.22 KB max page, 6.22 KB max dirty page
0x10d1bfc918e0:leaf: 32 pages, 90658.12 KB, 0/32 clean/dirty pages, 0.00 /90658.12 /90641.77 clean/dirty/updates KB, 8704.64 KB max page, 8704.64 KB max dirty page
0x10d1bfc918e0:file:WiredTigerShared.wt_stable(<live>):
0x10d1bfc918e0:internal: 1 pages, 0.85 KB, 1/0 clean/dirty pages, 0.85/0.00 clean / dirty KB, 0.85 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 1 pages, 13.40 KB, 1/0 clean/dirty pages, 13.40 /0.00 /6.95 clean/dirty/updates KB, 13.40 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:T00002.wt_stable(<live>):
0x10d1bfc918e0:internal: 224 pages, 7315.25 KB, 216/8 clean/dirty pages, 7135.38/179.87 clean / dirty KB, 165.29 KB max page, 32.02 KB max dirty page
0x10d1bfc918e0:leaf: 20666 pages, 596779.12 KB, 20666/0 clean/dirty pages, 596779.12 /0.00 /547.12 clean/dirty/updates KB, 262.13 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.12(<live>):
0x10d1bfc918e0:internal: 1 pages, 0.79 KB, 1/0 clean/dirty pages, 0.79/0.00 clean / dirty KB, 0.79 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 0 pages
0x10d1bfc918e0:file:WiredTigerSharedHS.wt_stable(<live>):
0x10d1bfc918e0:internal: 18 pages, 10040.49 KB, 18/0 clean/dirty pages, 10040.49/0.00 clean / dirty KB, 3595.34 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 8434 pages, 89652.36 KB, 8434/0 clean/dirty pages, 89652.36 /0.00 /0.00 clean/dirty/updates KB, 180.29 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:WiredTigerShared.wt_stable(<live>):
0x10d1bfc918e0:internal: 1 pages, 0.80 KB, 1/0 clean/dirty pages, 0.80/0.00 clean / dirty KB, 0.80 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 0 pages
0x10d1bfc918e0:file:T00001.wt(<live>):
0x10d1bfc918e0:internal: 52 pages, 55297.46 KB, 52/0 clean/dirty pages, 55297.46/0.00 clean / dirty KB, 1394.60 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 249323 pages, 1071692.23 KB, 81199/168124 clean/dirty pages, 193635.25 /878056.98 /561048.53 clean/dirty/updates KB, 1358.46 KB max page, 1358.46 KB max dirty page
0x10d1bfc918e0:file:WiredTigerSharedHS.wt_stable(<live>):
0x10d1bfc918e0:internal: 1 pages, 1330.70 KB, 1/0 clean/dirty pages, 1330.70/0.00 clean / dirty KB, 1330.70 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 46 pages, 315.95 KB, 46/0 clean/dirty pages, 315.95 /0.00 /0.00 clean/dirty/updates KB, 32.68 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:WiredTigerHS.wt(<live>):
0x10d1bfc918e0:internal: 5 pages, 11210.70 KB, 5/0 clean/dirty pages, 11210.70/0.00 clean / dirty KB, 3287.61 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 693 pages, 7911.50 KB, 693/0 clean/dirty pages, 7911.50 /0.00 /671.93 clean/dirty/updates KB, 411.87 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:file:WiredTiger.wt(<live>):
0x10d1bfc918e0:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:leaf: 1 pages, 34.74 KB, 1/0 clean/dirty pages, 34.74 /0.00 /20.54 clean/dirty/updates KB, 34.74 KB max page, 0.00 KB max dirty page
0x10d1bfc918e0:cache dump: total found: 2918.41 MB vs tracked inuse 2702.24 MB
0x10d1bfc918e0:total dirty bytes: 1285.67 MB vs tracked dirty 1285.67 MB

format-stress-test-disagg-switch-data-validation-stepdown-async-2 task_log

Logs:

[1785492043:076325][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 803: Error at src/btree/bt_curnext.c:803: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1785492043:076336][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
[1785492043:076339][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 98: Error at src/evict/evict_thread.c:98: "ret" failed: Connection timed out
[1785492043:076343][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: Connection timed out
[1785492043:076346][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1785492043:076349][33471:0xffffba7cdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

format-stress-test-disagg-switch-data-validation-stepdown-async-2 task_log

Logs:

#0  0x0000ffffbc8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffbc8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffbc87b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffbc862224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffbcc5ecbc [PAC] in __wt_abort (session=session@entry=0x10d1bfc918e0) at /data/mci/153c537ec09c90c75f45cb9a42fad28f/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffbcd0b8b0 in __wt_panic_func (session=session@entry=0x10d1bfc918e0, error=error@entry=110, func=func@entry=0xffffbce17630 <__PRETTY_FUNCTION__.6> "__evict_thread_run", line=line@entry=121, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffbcdb05a8 "eviction thread error") at /data/mci/153c537ec09c90c75f45cb9a42fad28f/wiredtiger/src/support/err.c:633
#5  0x0000ffffbcc169b0 in __evict_thread_run (session=0x10d1bfc918e0, thread=0x10d1bfe06910) at /data/mci/153c537ec09c90c75f45cb9a42fad28f/wiredtiger/src/evict/evict_thread.c:121
#6  0x0000ffffbcd2822c in __thread_run (arg=0x10d1bfe06910) at /data/mci/153c537ec09c90c75f45cb9a42fad28f/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffffbc8c2834 in start_thread () from /lib64/libc.so.6
#8  0x0000ffffbc866e5c [PAC] in thread_start () from /lib64/libc.so.6

Repro Artifacts:

CONFIG

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Screenshot 2026-08-01 at 5.36.50 pm.png
201 kB
Aug 01 2026 07:44:06 AM UTC

is related to

WT-18250 test/format (disagg) run more than 15 minutes past the maximum time: cache over dirty trigger, eviction server idle

Open

related to

WT-18250 test/format (disagg) run more than 15 minutes past the maximum time: cache over dirty trigger, eviction server idle

Open

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Aug 01 2026 07:31:14 AM UTC
Updated:: Aug 14 2026 06:00:04 AM UTC