Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools

Assigned Teams:

Storage Engines
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- amazon2023-disagg-stress
Failing Tasks:
- format-stress-test-disagg-switch-data-validation-2
First Failing Revision:
- e52fc07190d979182526f7d7444818e5a32cde9f(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-16957
Count of Linked BFGs (Last 30 days):
2

format-stress-test-disagg-switch-data-validation-2 on amazon2023-disagg-stress

Host: i-0d8a009d699a57a1a
Project: wiredtiger
Commit: 5379df24
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-data-validation-2

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

[1773816967:110985][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

0x110cf3d3bf60:transaction state dump
0x110cf3d3bf60:current ID: 675582
0x110cf3d3bf60:last running ID: 675582
0x110cf3d3bf60:metadata_pinned ID: 675582
0x110cf3d3bf60:oldest ID: 675582
0x110cf3d3bf60:durable timestamp: (0, 6803247)
0x110cf3d3bf60:oldest timestamp: (0, 6800492)
0x110cf3d3bf60:pinned timestamp: (0, 6800492)
0x110cf3d3bf60:stable timestamp: (0, 6802953)
0x110cf3d3bf60:has_durable_timestamp: yes
0x110cf3d3bf60:has_oldest_timestamp: yes
0x110cf3d3bf60:has_pinned_timestamp: yes
0x110cf3d3bf60:has_stable_timestamp: yes
0x110cf3d3bf60:oldest_is_pinned: yes
0x110cf3d3bf60:stable_is_pinned: no
0x110cf3d3bf60:checkpoint running: no
0x110cf3d3bf60:checkpoint generation: 1
0x110cf3d3bf60:checkpoint pinned ID: 0
0x110cf3d3bf60:checkpoint txn ID: 0
0x110cf3d3bf60:session count: 33
0x110cf3d3bf60:Transaction state of active sessions:
0x110cf3d3bf60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x110cf3d3bf60:cache dump
0x110cf3d3bf60:cache full: no
0x110cf3d3bf60:cache clean check: no (63.586%)
0x110cf3d3bf60:cache dirty check: yes (21.366%)
0x110cf3d3bf60:cache updates check: yes (19.878%)
0x110cf3d3bf60:file:T00002.wt_stable(<live>):
0x110cf3d3bf60:internal: 86 pages, 2374.86 KB, 4/82 clean/dirty pages, 257.62/2117.23 clean / dirty KB, 85.60 KB max page, 85.60 KB max dirty page
0x110cf3d3bf60:leaf: 7708 pages, 219503.60 KB, 7297/411 clean/dirty pages, 208470.10 /11033.49 /1089.93 clean/dirty/updates KB, 187.57 KB max page, 39.84 KB max dirty page
0x110cf3d3bf60:file:T00003.wt_stable(<live>):
0x110cf3d3bf60:internal: 103 pages, 3577.46 KB, 6/97 clean/dirty pages, 501.84/3075.62 clean / dirty KB, 162.93 KB max page, 162.93 KB max dirty page
0x110cf3d3bf60:leaf: 8478 pages, 236104.25 KB, 8067/411 clean/dirty pages, 225441.40 /10662.84 /989.64 clean/dirty/updates KB, 187.95 KB max page, 41.51 KB max dirty page
0x110cf3d3bf60:file:T00003.wt_stable/WiredTigerCheckpoint.9(<live>):
0x110cf3d3bf60:internal: 104 pages, 3449.20 KB, 104/0 clean/dirty pages, 3449.20/0.00 clean / dirty KB, 162.25 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:leaf: 9055 pages, 323829.49 KB, 9055/0 clean/dirty pages, 323829.49 /0.00 /0.00 clean/dirty/updates KB, 188.38 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:file:T00003.wt_ingest(<live>):
0x110cf3d3bf60:internal: 1 pages, 131.45 KB, 0/1 clean/dirty pages, 0.00/131.45 clean / dirty KB, 131.45 KB max page, 131.45 KB max dirty page
0x110cf3d3bf60:leaf: 753 pages, 290368.69 KB, 0/753 clean/dirty pages, 0.00 /290368.69 /289761.01 clean/dirty/updates KB, 1462.63 KB max page, 1462.63 KB max dirty page
0x110cf3d3bf60:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.8(<live>) eviction disabled at open:
0x110cf3d3bf60:internal: 1 pages, 0.47 KB, 1/0 clean/dirty pages, 0.47/0.00 clean / dirty KB, 0.47 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:leaf: 0 pages
0x110cf3d3bf60:file:T00002.wt_stable/WiredTigerCheckpoint.9(<live>):
0x110cf3d3bf60:internal: 86 pages, 2262.07 KB, 86/0 clean/dirty pages, 2262.07/0.00 clean / dirty KB, 85.09 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:leaf: 8330 pages, 307115.43 KB, 8330/0 clean/dirty pages, 307115.43 /0.00 /0.00 clean/dirty/updates KB, 188.44 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:file:T00002.wt_ingest(<live>):
0x110cf3d3bf60:internal: 1 pages, 13.50 KB, 0/1 clean/dirty pages, 0.00/13.50 clean / dirty KB, 13.50 KB max page, 13.50 KB max dirty page
0x110cf3d3bf60:leaf: 75 pages, 288682.21 KB, 0/75 clean/dirty pages, 0.00 /288682.21 /285155.99 clean/dirty/updates KB, 6532.79 KB max page, 6532.79 KB max dirty page
0x110cf3d3bf60:file:T00001.wt(<live>):
0x110cf3d3bf60:internal: 128 pages, 2270.64 KB, 7/121 clean/dirty pages, 664.74/1605.90 clean / dirty KB, 163.69 KB max page, 86.95 KB max dirty page
0x110cf3d3bf60:leaf: 3605 pages, 171557.72 KB, 3211/394 clean/dirty pages, 149964.47 /21593.25 /1982.74 clean/dirty/updates KB, 160.41 KB max page, 81.75 KB max dirty page
0x110cf3d3bf60:file:WiredTigerSharedHS.wt_stable(<live>):
0x110cf3d3bf60:internal: 1 pages, 4.25 KB, 0/1 clean/dirty pages, 0.00/4.25 clean / dirty KB, 4.25 KB max page, 4.25 KB max dirty page
0x110cf3d3bf60:leaf: 21 pages, 489.86 KB, 21/0 clean/dirty pages, 489.86 /0.00 /0.00 clean/dirty/updates KB, 32.22 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:file:WiredTigerShared.wt_stable(<live>):
0x110cf3d3bf60:internal: 1 pages, 0.53 KB, 1/0 clean/dirty pages, 0.53/0.00 clean / dirty KB, 0.53 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:leaf: 1 pages, 6.22 KB, 1/0 clean/dirty pages, 6.22 /0.00 /0.00 clean/dirty/updates KB, 6.22 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:file:WiredTigerHS.wt(<live>):
0x110cf3d3bf60:internal: 1 pages, 162.97 KB, 0/1 clean/dirty pages, 0.00/162.97 clean / dirty KB, 162.97 KB max page, 162.97 KB max dirty page
0x110cf3d3bf60:leaf: 2 pages, 170.79 KB, 2/0 clean/dirty pages, 170.79 /0.00 /0.00 clean/dirty/updates KB, 86.71 KB max page, 0.00 KB max dirty page
0x110cf3d3bf60:file:WiredTiger.wt(<live>):
0x110cf3d3bf60:internal: 1 pages, 0.77 KB, 0/1 clean/dirty pages, 0.00/0.77 clean / dirty KB, 0.77 KB max page, 0.77 KB max dirty page
0x110cf3d3bf60:leaf: 0 pages
0x110cf3d3bf60:cache dump: total found: 1953.36 MB vs tracked inuse 1808.67 MB
0x110cf3d3bf60:total dirty bytes: 614.70 MB vs tracked dirty 614.70 MB

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

[1773816967:122614][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122623][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122625][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3059: Error at src/evict/evict_lru.c:3059: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1773816967:122628][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122630][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122632][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122633][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122635][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122637][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122639][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122641][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3059: Error at src/evict/evict_lru.c:3059: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1773816967:122643][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122645][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122647][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3059: Error at src/evict/evict_lru.c:3059: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1773816967:122649][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122651][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122653][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122655][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122657][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122659][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122661][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122663][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122665][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3059: Error at src/evict/evict_lru.c:3059: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1773816967:122667][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122669][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122671][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122672][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122674][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122676][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122678][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122680][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122683][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122685][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122687][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122689][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122691][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 393: Error at src/evict/evict_page.c:393: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1773816967:122692][21605:0xffff80bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_page, 3059: Error at src/evict/evict_lru.c:3059: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)"

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

#0  0x0000ffff81cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff81cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff81c7b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff81c62224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff820396c0 [PAC] in __wt_abort (session=session@entry=0x110cf3d3bf60) at /data/mci/c6eae15cdd750c94da0b03fc0e292355/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff820de2a4 in __wt_panic_func (session=session@entry=0x110cf3d3bf60, error=error@entry=110, func=func@entry=0xffff821d4da0 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff82174158 "eviction thread error") at /data/mci/c6eae15cdd750c94da0b03fc0e292355/wiredtiger/src/support/err.c:611
#5  0x0000ffff81feebe8 in __evict_thread_run (session=0x110cf3d3bf60, thread=0x110d3deae280) at /data/mci/c6eae15cdd750c94da0b03fc0e292355/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffff820f9844 in __thread_run (arg=0x110d3deae280) at /data/mci/c6eae15cdd750c94da0b03fc0e292355/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffff81cc2834 in start_thread () from /lib64/libc.so.6
#8  0x0000ffff81c66e5c [PAC] in thread_start () from /lib64/libc.so.6

logs

Repro Artifacts:

CONFIG

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Mar 18 2026 10:14:35 AM UTC
Updated:: Mar 19 2026 12:08:04 AM UTC

Details

Description

Attachments

Activity

People

Dates