test/format (disagg mode=switch) ingest drain aborts on 'cbt->ins == NULL': stable btree contains inserts prior to draining

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-test-disagg-switch-pull-request-1 on ubuntu2004-asan

      Host: i-0eaa0442f6f2412c5
      Project: wiredtiger
      Commit: b012fbfc
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-pull-request-1 task_log

      Logs:

      [1787548394:517082][18093:0x7f8a7b39a740], t, file:T00002.wt_ingest, disagg-step-up: [WT_VERB_DEFAULT][ERROR]: int __layered_drain_worker_run(WT_SESSION_IMPL *, WT_THREAD *), 854: Failed to drain ingest and truncate list for "file:T00002.wt_ingest": WT_RESTART: restart the operation (internal)
[1787548394:526712][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_DEFAULT][ERROR]: void __layered_assert_stable_btree_state(WT_SESSION_IMPL *, WT_CURSOR_BTREE *, WT_UPDATE *), 34: WiredTiger assertion failed: 'cbt->ins == ((void*)0)'. The stable btree should not contain inserts prior to draining
[1787548394:526806][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 29: aborting WiredTiger library
[1787548394:526823][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526837][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526851][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526860][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526870][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526879][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526894][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found
[1787548394:526904][18093:0x7f8a35bbf700], t, file:T00003.wt_stable, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: int __curversion_next_single_key(WT_CURSOR *), 833: Error at src/cursor/cur_version.c:833: "__curversion_process_on_disk(cursor, tombstone, page, &upd_found, &done)" failed: WT_NOTFOUND: item not found

      logs

      format-stress-test-disagg-switch-pull-request-1 task_log

      Logs:

      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f8a7b3c1859 in __GI_abort () at abort.c:79
#2  0x00007f8a7c088042 in __wt_abort () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#3  0x00007f8a7bce9eaf in __layered_assert_stable_btree_state () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#4  0x00007f8a7bce87fe in __layered_move_updates () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#5  0x00007f8a7bce4c84 in __layered_copy_ingest_table () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#6  0x00007f8a7bce1a72 in __layered_drain_ingest_table_and_truncate_list () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#7  0x00007f8a7bcdce63 in __layered_drain_worker_run () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#8  0x00007f8a7c40f1ad in __thread_run () from /data/mci/5f3abf9a00817c67a815e81fd5eb917b/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0
#9  0x00007f8a7cbd8f97 in asan_thread_start(void*) () from /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.7Jt/lib/clang/19/lib/x86_64-mongodb-linux/libclang_rt.asan.so
#10 0x00007f8a7b599609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f8a7b4be353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

      logs

      Repro Artifacts:

            Assignee:
            Shoufu Du
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: