Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Foundations
Total Hours with Assigned Team:
68.303
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- ubuntu2004-arm64-nonstandalone
Failing Tasks:
- format-stress-test-disagg-switch-data-validation-2
First Failing Revision:
- b1e370ae684eb8a403e4714d1fa16a74b095f4f0(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-18114
Count of Linked BFGs (Last 30 days):
1

format-stress-test-disagg-switch-data-validation-2 on ubuntu2004-arm64-nonstandalone

Host: i-098459d832ca8af3a
Project: wiredtiger
Commit: b1e370ae
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-stress-test-disagg-switch-data-validation-2

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/products.sh: line 13: /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/installer.sh: No such file or directory

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

[440/680] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o
[448/680] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

[1784099797:809198][31657:0xfffdca222c80], t, connection-open-session: [WT_VERB_DEFAULT][ERROR]: __wti_disagg_set_crypt_key, 546: set_key timestamp 28473584 must be strictly greater than the stable timestamp 28474104: Invalid argument
[1784102712:363885][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wti_rec_row_leaf, 1190: WiredTiger assertion failed: 'vpack->cell == ((void *)0) || __wt_cell_type_raw(vpack->cell) != (11 << 4)'. Expression returned false
[1784102712:363917][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
[1784102712:363930][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363939][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363942][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363945][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:363948][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363951][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363954][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363956][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363959][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363961][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363964][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363966][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_evict_page, 220: Error at src/evict/evict_dispatch.c:220: "__evict_get_ref(session, is_server, &btree, &ref, &previous_state)" failed: WT_NOTFOUND: item not found
[1784102712:363970][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363972][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363975][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363977][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:363980][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363982][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:363989][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363991][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:363994][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363997][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:363999][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:364002][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364004][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_try_readlock, 151: Error at src/support/mtx_rw.c:151: "__wt_atomic_cas_uint64_v(&l->u.v, old.u.v, new.u.v) ? 0 : EBUSY" failed: Device or resource busy
[1784102712:364006][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364009][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364011][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364014][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364036][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364039][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364041][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364044][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364046][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364049][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364051][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource busy
[1784102712:364054][32885:0xffff8fba9c80], t, file:T00001.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_evict, 441: Error at src/evict/evict_page.c:441: "__evict_review(session, ref, flags, &inmem_split)" failed: Device or resource

logs

format-stress-test-disagg-switch-data-validation-2 task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffff9223daac in __GI_abort () at abort.c:79
#2  0x0000ffff9272a4bc in __wt_abort (session=session@entry=0x52b17f388100) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/os_common/os_abort.c:32
#3  0x0000ffff9276246c in __wti_rec_row_leaf (session=session@entry=0x52b17f388100, r=r@entry=0x52b17db02c80, pageref=pageref@entry=0x52b17d8577c0, salvage=salvage@entry=0x0) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/reconcile/rec_row.c:1190
#4  0x0000ffff9277a710 in __reconcile (session=session@entry=0x52b17f388100, ref=ref@entry=0x52b17d8577c0, salvage=salvage@entry=0x0, flags=flags@entry=680, page_lockedp=page_lockedp@entry=0xffff8fba7caf) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/reconcile/rec_write.c:309
#5  0x0000ffff92779898 in __wt_reconcile (session=session@entry=0x52b17f388100, ref=ref@entry=0x52b17d8577c0, salvage=salvage@entry=0x0, flags=flags@entry=680) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/reconcile/rec_write.c:129
#6  0x0000ffff926dab08 in __evict_reconcile (session=session@entry=0x52b17f388100, ref=ref@entry=0x52b17d8577c0, evict_flags=evict_flags@entry=0) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_page.c:1370
#7  0x0000ffff926dd348 in __wt_evict (session=session@entry=0x52b17f388100, ref=0x52b17d8577c0, previous_state=3 '\003', flags=flags@entry=0) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_page.c:490
#8  0x0000ffff926d1ca8 in __wti_evict_page (session=session@entry=0x52b17f388100, is_server=is_server@entry=true) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_dispatch.c:254
#9  0x0000ffff926de17c in __wti_evict_lru_pages (session=session@entry=0x52b17f388100, is_server=is_server@entry=true) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_queue.c:140
#10 0x0000ffff926e1c00 in __evict_pass (session=session@entry=0x52b17f388100) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_thread.c:696
#11 0x0000ffff926e1d44 in __evict_server (session=session@entry=0x52b17f388100, did_work=did_work@entry=0xffff8fba901f) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_thread.c:198
#12 0x0000ffff926e21dc in __evict_thread_run (session=0x52b17f388100, thread=0x52b17fe0afa0) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/evict/evict_thread.c:93
#13 0x0000ffff927f3a60 in __thread_run (arg=0x52b17fe0afa0) at /data/mci/c557327d6e021e6f5b606b55bddb0c44/wiredtiger/src/support/thread_group.c:32
#14 0x0000ffff92397624 in start_thread (arg=0xffff927f3a0c <__thread_run>) at pthread_create.c:477
#15 0x0000ffff922ee66c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

logs

Repro Artifacts:

CONFIG

is related to

WT-18055 __wti_rec_row_leaf assertion failed: vpack->cell == ((void *)0) || __wt_cell_type_raw(vpack->cell) != (11 << 4)

Closed

WT-18108 task-timed-out: format-stress-test-disagg-switch: Invalid argument timestamp must be strictly greater than the stable timestamp

Closed

Assignee:: [DO NOT USE] Backlog - Storage Engines Team
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 2 Start watching this issue

Created:: Jul 16 2026 02:20:33 AM UTC
Updated:: Jul 16 2026 09:02:26 AM UTC

Details

Description

Attachments

Issue Links

Activity

People

Dates