-
Type:
Build Failure
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Storage Engines, Storage Engines - Foundations
-
220.634
-
None
-
None
-
1
format-stress-test-disagg-leader-pull-request-2 on ubuntu2004-release
Host: i-04566e82e0afd5e6c
Project: wiredtiger
Commit: e428a613
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-leader-pull-request-2 task_log
Logs:
[1778076856:860376][23761:0x7fc203b8f700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:844: Connection::db_statement(size_t)::<lambda(sqlite3_stmt*)>: sqlite3_reset; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140443149231000'
[1778076856:865775][23761:0x7fc204390700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:844: Connection::db_statement(size_t)::<lambda(sqlite3_stmt*)>: sqlite3_reset; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140444357057176'
[1778076856:872446][23761:0x7fc20338e700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:844: Connection::db_statement(size_t)::<lambda(sqlite3_stmt*)>: sqlite3_reset; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140446907364360'
[1778076856:873353][23761:0x7fc203b8f700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:1645: void Pages::put(uint64_t, uint64_t, uint64_t, WT_PAGE_LOG_PUT_ARGS*, const WT_ITEM*): sqlite3_step; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140443152064968'
[1778076856:876852][23761:0x7fc203b8f700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:844: Connection::db_statement(size_t)::<lambda(sqlite3_stmt*)>: sqlite3_reset; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140443152064968'
[1778076856:876885][23761:0x7fc203b8f700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:587: int safe_call(WT_SESSION*, S*, MemberFunc, Args&& ...) [with T = PaliteHandle; S = __wt_page_log_handle; MemberFunc = int (PaliteHandle::*)(long unsigned int, long unsigned int, __wt_page_log_put_args*, const __wt_item*); Args = {long unsigned int&, long unsigned int&, __wt_page_log_put_args*&, const __wt_item*&}; WT_SESSION = __wt_session]: System error: sqlite3_step; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140443152064968': No space left on device
[1778076856:876934][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write_internal, 186: Error at src/block_disagg/block_disagg_write.c:186: "plhandle->plh_put(plhandle, &session->iface, page_id, 0, &put_args, buf)" failed: No space left on device
[1778076856:877758][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write, 246: Error at src/block_disagg/block_disagg_write.c:246: "__wti_block_disagg_write_internal(session, block_disagg, buf, block_meta, page_image_size, &size, &checksum, data_checksum, checkpoint_io)" failed: No space left on device
[1778076856:877981][23761:0x7fc204390700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:1645: void Pages::put(uint64_t, uint64_t, uint64_t, WT_PAGE_LOG_PUT_ARGS*, const WT_ITEM*): sqlite3_step; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140444358458632'
[1778076856:878021][23761:0x7fc204390700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:844: Connection::db_statement(size_t)::<lambda(sqlite3_stmt*)>: sqlite3_reset; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140444358458632'
[1778076856:878042][23761:0x7fc204390700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:587: int safe_call(WT_SESSION*, S*, MemberFunc, Args&& ...) [with T = PaliteHandle; S = __wt_page_log_handle; MemberFunc = int (PaliteHandle::*)(long unsigned int, long unsigned int, __wt_page_log_put_args*, const __wt_item*); Args = {long unsigned int&, long unsigned int&, __wt_page_log_put_args*&, const __wt_item*&}; WT_SESSION = __wt_session]: System error: sqlite3_step; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140444358458632': No space left on device
[1778076856:877766][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_blkcache_write, 843: Error at src/block_cache/block_io.c:843: "checkpoint ? bm->checkpoint(bm, session, ip, block_meta, btree->ckpt, data_checksum) : bm->write(bm, session, ip, block_meta, page_image_size, addr, addr_sizep, data_checksum, checkpoint_io)" failed: No space left on device
[1778076856:878058][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write, 1054: Error at src/reconcile/rec_write.c:1054: "__wt_blkcache_write(session, buf, block_meta, buf->size, addr, addr_sizep, compressed_sizep, checkpoint, checkpoint_io, compressed)" failed: No space left on device
[1778076856:878067][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write_image, 2292: Error at src/reconcile/rec_write.c:2292: "__rec_write(session, &chunk->image, multi->block_meta, addr, addr_sizep, compressed_sizep, false, F_ISSET(r, WT_REC_CHECKPOINT), false)" failed: No space left on device
[1778076856:878063][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write_internal, 186: Error at src/block_disagg/block_disagg_write.c:186: "plhandle->plh_put(plhandle, &session->iface, page_id, 0, &put_args, buf)" failed: No space left on device
[1778076856:878072][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2574: Error at src/reconcile/rec_write.c:2574: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: No space left on device
[1778076856:878087][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wti_block_disagg_write, 246: Error at src/block_disagg/block_disagg_write.c:246: "__wti_block_disagg_write_internal(session, block_disagg, buf, block_meta, page_image_size, &size, &checksum, data_checksum, checkpoint_io)" failed: No space left on device
[1778076856:878093][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_blkcache_write, 843: Error at src/block_cache/block_io.c:843: "checkpoint ? bm->checkpoint(bm, session, ip, block_meta, btree->ckpt, data_checksum) : bm->write(bm, session, ip, block_meta, page_image_size, addr, addr_sizep, data_checksum, checkpoint_io)" failed: No space left on device
[1778076856:878098][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write, 1054: Error at src/reconcile/rec_write.c:1054: "__wt_blkcache_write(session, buf, block_meta, buf->size, addr, addr_sizep, compressed_sizep, checkpoint, checkpoint_io, compressed)" failed: No space left on device
[1778076856:878103][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_write_image, 2292: Error at src/reconcile/rec_write.c:2292: "__rec_write(session, &chunk->image, multi->block_meta, addr, addr_sizep, compressed_sizep, false, F_ISSET(r, WT_REC_CHECKPOINT), false)" failed: No space left on device
[1778076856:878107][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __rec_split_write, 2574: Error at src/reconcile/rec_write.c:2574: "__rec_write_image(session, r, chunk, addr, &addr_size, &compressed_size, last_block)" failed: No space left on device
[1778076856:878084][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_parallel_thread_run, 237: Error at src/checkpoint/checkpoint_parallel.c:237: "ret" failed: No space left on device
[1778076856:878113][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __checkpoint_parallel_thread_run, 237: Error at src/checkpoint/checkpoint_parallel.c:237: "ret" failed: No space left on device
[1778076856:878606][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: No space left on device
[1778076856:878596][23761:0x7fc203b8f700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: No space left on device
[1778076856:878988][23761:0x7fc20338e700], t, file:T00003.wt_stable, checkpoint-page-reconciliation-threads: [WT_VERB_EXTENSION][ERROR]: ext/page_log/palite/palite.cpp:1645: void Pages::put(uint64_t, uint64_t, uint64_t, WT_PAGE_LOG_PUT_ARGS*, const WT_ITEM*): sqlite3_step; return: 13 (database or disk is full); details: 13 (database or disk is full); args: '140446908117816'
[1778076856:879002][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_DEFAULT][ERROR]: __checkpoint_parallel_thread_stop, 268: Checkpoint page reconciliation thread 0 stopping while a transaction is running: Input/output error
[1778076856:879013][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_DEFAULT][ERROR]: __checkpoint_parallel_thread_stop, 268: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1778076856:879022][23761:0x7fc204390700], t, checkpoint-page-reconciliation-threads: [WT_VERB_ERROR_RETURNS][ERROR
format-stress-test-disagg-leader-pull-request-2 task_log
Logs:
Command 'shell.exec' in function 'format test disagg' (step 3 of 3) stopped early: context canceled.
format-stress-test-disagg-leader-pull-request-2 task_log
Logs:
Running task commands failed: running command: command stopped early: context canceled
format-stress-test-disagg-leader-pull-request-2 task_log
Logs:
Command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' failed: opening target archive file '/data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger/cmake_build/stat_files.tar.gz': creating file '/data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger/cmake_build/stat_files.tar.gz': open /data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger/cmake_build/stat_files.tar.gz: no space left on device. Command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' failed: opening target archive file '/data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger.tgz': creating file '/data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger.tgz': open /data/mci/7806656ace297c37dc08d2f73d095afc/wiredtiger.tgz: no space left on device. Command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' failed: opening target archive file '/data/mci/7806656ace297c37dc08d2f73d095afc/wt-hang-analyzer.tgz': creating file '/data/mci/7806656ace297c37dc08d2f73d095afc/wt-hang-analyzer.tgz': open /data/mci/7806656ace297c37dc08d2f73d095afc/wt-hang-analyzer.tgz: no space left on device.
Repro Artifacts: