Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Persistence
Total Hours with Assigned Team:
24.76
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
- ubuntu2004-arm64-nonstandalone
Failing Tasks:
- unit-test-extra-long
First Failing Revision:
- 02a9310d508af20a4146025176c35297d3712119(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-18669
Count of Linked BFGs (Last 30 days):
1

unit-test-extra-long on ubuntu2004-arm64-nonstandalone

Host: i-03c4650f899d8ebd1
Project: wiredtiger
Commit: 02a9310d
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task unit-test-extra-long

unit-test-extra-long task_log

Logs:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x0000ffffb54f0aac in __GI_abort () at abort.c:79
#2  0x0000ffffb49d2618 in __wt_abort (session=session@entry=0x3028b5b8a0c0) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/os_common/os_abort.c:32
#3  0x0000ffffb4a80930 in __wt_panic_func (session=session@entry=0x3028b5b8a0c0, error=error@entry=16, func=func@entry=0xffffb4b883c0 <__PRETTY_FUNCTION__.39> "__checkpoint_db_internal", line=line@entry=2250, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffb4af4fe8 "Disaggregated storage checkpoint failed, unable to rollback, panic to avoid corruption") at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/support/err.c:633
#4  0x0000ffffb48b00c0 in __checkpoint_db_internal (session=session@entry=0x3028b5b8a0c0, cfg=cfg@entry=0xffffd70e7aa0) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/checkpoint/checkpoint_txn.c:2250
#5  0x0000ffffb48b0400 in __checkpoint_db_wrapper (session=session@entry=0x3028b5b8a0c0, cfg=cfg@entry=0xffffd70e7aa0) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/checkpoint/checkpoint_txn.c:2309
#6  0x0000ffffb48b0904 in __wt_checkpoint_db (session=0x3028b5b8a0c0, cfg=cfg@entry=0xffffd70e7aa0, waiting=waiting@entry=true) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/checkpoint/checkpoint_txn.c:2406
#7  0x0000ffffb4ab27c8 in __wt_txn_global_shutdown (session=session@entry=0x3028b5b84000, cfg=cfg@entry=0xffffd70e7b68) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/txn/txn.c:2881
#8  0x0000ffffb48c8314 in __conn_close (wt_conn=0x3028ba3a2800, config=<optimized out>) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/src/conn/conn_api.c:1377
#9  0x0000ffffb4c65ed8 in _wrap_Connection_close (self=<optimized out>, args=<optimized out>) at /data/mci/9faa7a7980cab7b6b359e0c3a5fa2c4c/wiredtiger/cmake_build/lang/python/CMakeFiles/wiredtiger_python.dir/wiredtigerPYTHON_wrap.c:8363
#10 0x0000ffffb587278c in cfunction_call (func=0xffffb4e05800, args=<optimized out>, kwargs=<optimized out>) at ../src/Python-3.10.4/Objects/methodobject.c:552
#11 0x0000ffffb588244c in _PyObject_Call (tstate=0x3028bfe127e0, callable=0xffffb4e05800, args=0xffffa58c4610, kwargs=<optimized out>) at ../src/Python-3.10.4/Objects/call.c:305
#12 PyObject_Call (callable=0xffffb4e05800, args=0xffffa58c4610, kwargs=<optimized out>) at ../src/Python-3.10.4/Objects/call.c:317
#13 0x0000ffffb5861150 in do_call_core (tstate=<optimized out>, trace_info=0xffffd70e81f0, func=0xffffb4e05800, callargs=0xffffa58c4610, kwdict=0x0) at ../src/Python-3.10.4/Python/ceval.c:5919
#14 _PyEval_EvalFrameDefault (tstate=0x3028bfe127e0, f=<optimized out>, throwflag=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:4277
#15 0x0000ffffb5872fcc in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052
#16 _PyEval_Vector (tstate=0x3028bfe127e0, con=0xffffb4e41d90, locals=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:5065
#17 _PyFunction_Vectorcall (func=0xffffb4e41d80, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Objects/call.c:342
#18 0x0000ffffb585f070 in _PyObject_VectorcallTstate (tstate=<optimized out>, callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123
#19 PyObject_Vectorcall (callable=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at ../src/Python-3.10.4/Include/cpython/abstract.h:123
#20 call_function (tstate=<optimized out>, trace_info=0xffffd70e83c0, pp_stack=<synthetic pointer>, oparg=<optimized out>, kwnames=0x0) at ../src/Python-3.10.4/Python/ceval.c:5867
#21 _PyEval_EvalFrameDefault (tstate=0x3028bfe127e0, f=<optimized out>, throwflag=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:4198
#22 0x0000ffffb5872fcc in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052
#23 _PyEval_Vector (tstate=0x3028bfe127e0, con=0xffffb4ebfb60, locals=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:5065
#24 _PyFunction_Vectorcall (func=0xffffb4ebfb50, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Objects/call.c:342
#25 0x0000ffffb5861150 in do_call_core (tstate=<optimized out>, trace_info=0xffffd70e8590, func=0xffffb4ebfb50, callargs=0xffffaf7a0790, kwdict=0xffffa3ec0980) at ../src/Python-3.10.4/Python/ceval.c:5919
#26 _PyEval_EvalFrameDefault (tstate=0x3028bfe127e0, f=<optimized out>, throwflag=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:4277
#27 0x0000ffffb5872fcc in _PyEval_EvalFrame (tstate=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, f=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>, throwflag=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at ../src/Python-3.10.4/Python/ceval.c:5052
#28 _PyEval_Vector (tstate=0x3028bfe127e0, con=0xffffb4ebfbf0, locals=0x0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Python/ceval.c:5065
#29 _PyFunction_Vectorcall (func=0xffffb4ebfbe0, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at ../src/Python-3.10.4/Objects/call.c:342

logs

unit-test-extra-long task_log

Logs:

[1789456504:079058][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerSharedHS.wt_stable, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __checkpoint_tree, 3365: checkpoint failed during cache flush and reconciliation: Device or resource busy
[1789456504:079118][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerSharedHS.wt_stable, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2793: Error at src/reconcile/rec_write.c:2793: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: Device or resource busy
[1789456504:079133][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerSharedHS.wt_stable, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_sync_file, 554: Error at src/btree/bt_sync.c:554: "__wt_reconcile(session, walk, NULL, __sync_page_rec_flags(session, page, rec_flags, checkpoint_scrub), NULL)" failed: Device or resource busy
[1789456504:079145][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerSharedHS.wt_stable, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_tree, 3365: Error at src/checkpoint/checkpoint_txn.c:3365: "ret" failed: Device or resource busy
[1789456504:079179][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 326: Error at src/cursor/cur_file.c:326: "ret" failed: WT_NOTFOUND: item not found
[1789456504:079192][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 326: Error at src/cursor/cur_file.c:326: "ret" failed: WT_NOTFOUND: item not found
[1789456504:079203][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write, 250: Error at src/block_disagg/block_disagg_write.c:250: "__wti_block_disagg_write_internal(session, block_disagg, buf, block_meta, page_image_size, &size, &checksum, data_checksum, checkpoint_io)" failed: Device or resource busy
[1789456504:079214][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_blkcache_write, 862: Error at src/block_cache/block_io.c:862: "checkpoint ? bm->checkpoint(bm, session, ip, block_meta, btree->ckpt, data_checksum) : bm->write(bm, session, ip, block_meta, page_image_size, addr, addr_sizep, data_checksum, checkpoint_io)" failed: Device or resource busy
[1789456504:079226][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write, 1165: Error at src/reconcile/rec_write.c:1165: "__wt_blkcache_write(session, buf, block_meta, buf->size, addr, addr_sizep, compressed_sizep, checkpoint, checkpoint_io, compressed)" failed: Device or resource busy
[1789456504:079236][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write_image, 2478: Error at src/reconcile/rec_write.c:2478: "__rec_write(session, &chunk->image, multi->block_meta, addr, addr_sizep, compressed_sizep, false, F_ISSET(r, WT_REC_CHECKPOINT), false)" failed: Device or resource busy
[1789456504:079246][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2793: Error at src/reconcile/rec_write.c:2793: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: Device or resource busy
[1789456504:079261][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_sync_file, 554: Error at src/btree/bt_sync.c:554: "__wt_reconcile(session, walk, NULL, __sync_page_rec_flags(session, page, rec_flags, checkpoint_scrub), NULL)" failed: Device or resource busy
[1789456504:079271][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_tree, 3365: Error at src/checkpoint/checkpoint_txn.c:3365: "ret" failed: Device or resource busy
[1789456504:079299][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_hs, 1704: Error at src/checkpoint/checkpoint_txn.c:1704: "ret" failed: Device or resource busy
[1789456504:079311][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_db_internal, 2046: Error at src/checkpoint/checkpoint_txn.c:2046: "__checkpoint_hs(session, cfg, hs_dhandle, hs_dhandle_shared)" failed: Device or resource busy
[1789456504:079321][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __checkpoint_db_internal, 2250: Disaggregated storage checkpoint failed, unable to rollback, panic to avoid corruption: Device or resource busy
[1789456504:079331][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __checkpoint_db_internal, 2250: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1789456504:079345][11877:0xffffb6be57c0], test_disagg_checkpoint_size18.test_disagg_checkpoint_size18.test_rec_write_err_pre_page_log_delta_failure, file:WiredTigerHS.wt, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

logs

Assignee:: Albert Song
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 1 Start watching this issue

Created:: Sep 16 2026 01:58:30 AM UTC
Updated:: Sep 16 2026 02:53:02 AM UTC

Details

Description

Attachments

Activity

People

Dates