-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
format-stress-test-disagg-leader-1 on ubuntu2004-stress-nonstandalone
Host: i-02df53a67d8bff98b
Project: wiredtiger
Commit: 67e89867
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-leader-1 task_log
Logs:
[1773738053:454187][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __schema_layered_worker_verify, 128: WiredTiger assertion failed: 'ingest_ret != 16'. Verify: file:T00002.wt_ingest ingest table on leader cannot be verified. Ingest contains dirty content or open cursors, which is an invalid state. [1773738053:454212][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library [1773738053:454219][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_checkpoint_close, 3040: Error at src/checkpoint/checkpoint_txn.c:3040: "ret" failed with WT_DIRTY_DATA: Table has dirty data: Device or resource busy [1773738053:454229][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_close, 471: Error at src/conn/conn_dhandle.c:471: "__wt_checkpoint_close(session, final)" failed: Device or resource busy [1773738053:454232][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_close, 473: Error at src/conn/conn_dhandle.c:473: "ret" failed: Device or resource busy [1773738053:454235][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_close_all, 896: Error at src/conn/conn_dhandle.c:896: "__conn_dhandle_close_one(session, uri, NULL, removed, mark_dead, check_visibility)" failed: Device or resource busy [1773738053:454239][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_execute_handle_operation, 28: Error at src/schema/schema_worker.c:28: "ret" failed: Device or resource busy [1773738053:454241][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_schema_worker, 191: Error at src/schema/schema_worker.c:191: "__wti_execute_handle_operation(session, uri, file_func, cfg, open_flags)" failed: Device or resource busy [1773738053:454245][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_session_get_dhandle, 939: Error at src/session/session_dhandle.c:939: "__wt_session_lock_dhandle(session, flags, &is_dead)" failed: Device or resource busy [1773738053:454247][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_dhandle_close_one, 845: Error at src/conn/conn_dhandle.c:845: "__wt_session_get_dhandle( session, uri, checkpoint, NULL, WT_DHANDLE_EXCLUSIVE | WT_DHANDLE_LOCK_ONLY)" failed: Device or resource busy [1773738053:454250][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_close_all, 896: Error at src/conn/conn_dhandle.c:896: "__conn_dhandle_close_one(session, uri, NULL, removed, mark_dead, check_visibility)" failed: Device or resource busy [1773738053:454253][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_execute_handle_operation, 28: Error at src/schema/schema_worker.c:28: "ret" failed: Device or resource busy [1773738053:454256][18689:0x7f51f251ac40], t, layered:T00002, WT_SESSION.verify: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_schema_worker, 191: Error at src/schema/schema_worker.c:191: "__wti_execute_handle_operation(session, uri, file_func, cfg, open_flags)" failed: Device or resource busy
format-stress-test-disagg-leader-1 task_log
Logs:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x00007f51f2541859 in __GI_abort () at abort.c:79 #2 0x00007f51f2b1c3fb in __wt_abort (session=session@entry=0x52363fc934b8) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/src/os_common/os_abort.c:32 #3 0x00007f51f2b91064 in __schema_layered_worker_verify (session=session@entry=0x52363fc934b8, uri=uri@entry=0x52377f01cb10 "layered:T00002", file_func=file_func@entry=0x7f51f29cfa62 <__wt_verify>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x7ffe2f60a3e0, open_flags=open_flags@entry=16777248) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/src/schema/schema_worker.c:128 #4 0x00007f51f2b9092b in __wt_schema_worker (session=0x52363fc934b8, uri=0x52377f01cb10 "layered:T00002", file_func=0x7f51f29cfa62 <__wt_verify>, name_func=0x0, cfg=0x7ffe2f60a3e0, open_flags=16777248) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/src/schema/schema_worker.c:247 #5 0x00007f51f2b907b5 in __wt_schema_worker (session=session@entry=0x52363fc934b8, uri=uri@entry=0x52363fc18004 "table:T00002", file_func=0x7f51f29cfa62 <__wt_verify>, name_func=name_func@entry=0x0, cfg=cfg@entry=0x7ffe2f60a3e0, open_flags=open_flags@entry=16777248) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/src/schema/schema_worker.c:225 #6 0x00007f51f2b94ef6 in __session_verify (wt_session=0x52363fc934b8, uri=0x52363fc18004 "table:T00002", config=<optimized out>) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/src/session/session_api.c:1817 #7 0x000000000041bb6b in table_verify (table=0x52363fc18000, arg=0x52363fc22f00) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/test/format/verify.c:49 #8 0x000000000041baf9 in tables_apply (func=func@entry=0x41bb0d <table_verify>, arg=arg@entry=0x52363fc22f00) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/test/format/format_inline.h:186 #9 0x000000000041d0d6 in wts_verify (conn=0x52363fc22f00, mirror_check=mirror_check@entry=true) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/test/format/verify.c:387 #10 0x000000000041b2ed in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/6dc332c9760b85c42df9669ef6b6096b/wiredtiger/test/format/t.c:426
Repro Artifacts: