Loading...

XML

Word

Printable

JSON

Type: Build Failure
Resolution: Unresolved
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Test Format
Labels:
- BB-Tools

Assigned Teams:

Storage Engines - Foundations
Total Hours with Assigned Team:
2.103
Sprint:
None
Story Points:
None
Evergreen Project:
- wiredtiger
Failure Type:
- failed
Failing Buildvariants:
Failing Tasks:
- format-failure-configs-test
First Failing Revision:
- b013d1d844aae30be24aa1f19cc765d5b57c16dd(wiredtiger)
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-19047
Count of Linked BFGs (Last 30 days):
7

Problem and workaround

format-failure-configs-test started failing on Amazon2023 ARM64 and Ubuntu2004 after b013d1d844 introduced the debug.cursor_copy configuration option.

When the option is absent from CONFIG.disagg.WT-18885, format consumes an extra configuration RNG draw even though cursor-copy remains disabled. This changes the saved config's initial role from follower to leader, exposing an existing mismatch: format skips precise-checkpoint timestamp initialization while the DisAgg helper enables precise checkpoints on the connection. The initial leader connection close then fails with "Precise checkpoint requires a stable timestamp".

The fix for this build failure is to add debug.cursor_copy=0 to the saved config. This avoids the extra RNG draw and restores the previous startup behavior, keeping the cache, workload, and role-switching scenario unchanged.

This ticket covers that workaround, not a fix to precise-checkpoint configuration. The broader configuration problem is tracked in WT-19048.

Local validation reproduced the failure without the pin and completed the pinned config successfully in 276 seconds. An Evergreen patch will validate format-failure-configs-test on both affected variants.

format-failure-configs-test on amazon2023-arm64

Host: i-0df7b385c4a823a9e
Project: wiredtiger
Commit: b013d1d8
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

Task Logs:

task format-failure-configs-test

format-failure-configs-test system_log

Logs:

CPU constraint detected (sustained >90% for 5+ minutes)

format-failure-configs-test task_log

Logs:

./run_format_configs.sh: line 38: 18168 Segmentation fault      ./t -1 -c $config -h WT_TEST_$basename_config > $log 2>&1

format-failure-configs-test task_log

Logs:

CONFIG.WT-17043: t: aborting to test recovery

format-failure-configs-test task_log

Logs:

CONFIG.disagg.WT-18885: [1794290354:225520][34556:0xffff8d9a4040], t, close_ckpt: [WT_VERB_DEFAULT][ERROR]: __checkpoint_prepare, 962: Precise checkpoint requires a stable timestamp: Invalid argument
CONFIG.disagg.WT-18885: [1794290354:225543][34556:0xffff8d9a4040], t, WT_CONNECTION.close: [WT_VERB_DEFAULT][ERROR]: __conn_close, 1265: failure during close, disabling further writes: Invalid argument
CONFIG.disagg.WT-18885: [1794290354:226046][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_close, 615: Error at src/cursor/cur_file.c:615: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226055][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_curhs_cache, 169: Error at src/cursor/cur_hs.c:169: "cursor->close(cursor)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226058][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 83: Error at src/evict/evict_thread.c:83: "__wt_curhs_cache(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226060][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226062][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226071][34556:0xffff89fcdbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: Unrecoverable utility thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226094][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_close, 615: Error at src/cursor/cur_file.c:615: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226099][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_curhs_cache, 169: Error at src/cursor/cur_hs.c:169: "cursor->close(cursor)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226101][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 83: Error at src/evict/evict_thread.c:83: "__wt_curhs_cache(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226103][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226105][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226108][34556:0xffff897bdbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: Unrecoverable utility thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226126][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_close, 615: Error at src/cursor/cur_file.c:615: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226131][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_curhs_cache, 169: Error at src/cursor/cur_hs.c:169: "cursor->close(cursor)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226134][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 83: Error at src/evict/evict_thread.c:83: "__wt_curhs_cache(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226135][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226137][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226144][34556:0xffff88fadbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: Unrecoverable utility thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226220][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_close, 615: Error at src/cursor/cur_file.c:615: "WT_SESSION_CHECK_PANIC(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226226][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_curhs_cache, 169: Error at src/cursor/cur_hs.c:169: "cursor->close(cursor)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226228][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 83: Error at src/evict/evict_thread.c:83: "__wt_curhs_cache(session)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226230][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226232][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: WT_PANIC: WiredTiger library panic
CONFIG.disagg.WT-18885: [1794290354:226234][34556:0xffff8a7ddbc0], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: Unrecoverable utility thread error: WT_PANIC: WiredTiger library panic

format-failure-configs-test task_log

Logs:

CONFIG.disagg.WT-18885: t: FAILED: wts_create_database/738: conn->close(conn, NULL): WT_PANIC: WiredTiger library panic

format-failure-configs-test task_log

Logs:

CONFIG.disagg.WT-18885: t: process aborting
CONFIG.disagg.WT-18885: WiredTiger Error: __wt_abort, 29: aborting WiredTiger library
CONFIG.disagg.WT-18885: Error at src/cursor/cur_stat.c:275: "WT_SESSION_CHECK_PANIC(session)" failed with WT_PANIC: WiredTiger library panic (-31804)
CONFIG.disagg.WT-18885: Error at src/conn/conn_stat.c:397: "ret" failed with WT_PANIC: WiredTiger library panic (-31804)
CONFIG.disagg.WT-18885: Error at src/cursor/cur_stat.c:426: "WT_SESSION_CHECK_PANIC(session)" failed with WT_PANIC: WiredTiger library panic (-31804)

format-failure-configs-test task_log

Logs:

#0  0x0000ffff8c0c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff8c0c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff8c07b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff8c062224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8c46ab2c [PAC] in __wt_abort (session=session@entry=0x0) at /data/mci/10dee3ada38ce242a4961fdd6b85c216/wiredtiger/src/os_common/os_abort.c:32
#4  0x000000000042a624 in testutil_die (e=-31804, fmt=fmt@entry=0x42bdb0 "%s/%d: %s") at /data/mci/10dee3ada38ce242a4961fdd6b85c216/wiredtiger/test/utility/misc.c:75
#5  0x0000000000426370 in wts_create_database () at /data/mci/10dee3ada38ce242a4961fdd6b85c216/wiredtiger/test/format/wts.c:738
#6  0x000000000041fe88 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/10dee3ada38ce242a4961fdd6b85c216/wiredtiger/test/format/t.c:395

is related to

WT-19048 Unify precise_checkpoint configuration for DisAgg in test/format

Open

WT-17043 Data race involving importing transaction snapshots

Closed

WT-18885 Victim cache serves empty skip-write images, tripping the empty-page assertion in __random_leaf_disk

Closed

Assignee:: Ivan Kochin
Reporter:: xgen-buildbaron-user
Votes:: 0 Vote for this issue
Watchers:: 3 Start watching this issue

Created:: Nov 10 2026 11:42:24 PM UTC
Updated:: Nov 11 2026 12:20:16 AM UTC