failed: format-stress-test-disagg-switch-2 on amazon2023-stress-tests-arm64 [wiredtiger @ 5ac4fcce]

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction

      format-stress-test-disagg-switch-2 on amazon2023-stress-tests-arm64

      Host: i-0551dcd056709a9db
      Project: wiredtiger
      Commit: 5ac4fcce
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-2 task_log

      Logs:

      /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/products.sh: line 13: /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/installer.sh: No such file or directory

      logs

      format-stress-test-disagg-switch-2 task_log

      Logs:

      [434/679] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o
[440/679] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic

      logs

      format-stress-test-disagg-switch-2 task_log

      Logs:

      [1783925045:132026][37244:0xffffb7f6f040], t, file:T00001.wt_ingest, disagg-step-up: [WT_VERB_DEFAULT][ERROR]: __layered_drain_worker_run, 831: Failed to clear ingest table "file:T00001.wt_ingest": WT_ROLLBACK: conflict between concurrent operations
[1783925045:213030][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_DEFAULT][ERROR]: __layered_drain_worker_run, 831: Failed to clear ingest table "file:T00003.wt_ingest": WT_ROLLBACK: conflict between concurrent operations
[1783925045:213065][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213069][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213072][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213074][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1783925045:213076][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213078][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213080][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1783925045:213082][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213084][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213085][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213087][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213089][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213091][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213093][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1783925045:213095][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213097][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213098][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213100][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213102][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213104][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213106][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213108][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213110][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1783925045:213112][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213113][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213115][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213117][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213119][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213121][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213126][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
[1783925045:213128][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213129][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213131][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213133][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213135][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213137][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213138][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
[1783925045:213140][37244:0xffffa136dbc0], t, file:T00003.wt_ingest, disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_help

      logs

      format-stress-test-disagg-switch-2 task_log

      Logs:

      #0  0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffffb667b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffffb6662224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffffb6a5acd0 [PAC] in __wt_abort (session=session@entry=0x53ebffc96aa0) at /data/mci/85e522c0f0c3685c4ab1799e0ea99aec/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffffb6b079d0 in __wt_panic_func (session=session@entry=0x53ebffc96aa0, error=error@entry=-31800, func=func@entry=0xffffb6c198b0 <__PRETTY_FUNCTION__.7> "__thread_run", line=line@entry=47, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffb6bee7e8 "Unrecoverable utility thread error") at /data/mci/85e522c0f0c3685c4ab1799e0ea99aec/wiredtiger/src/support/err.c:633
#5  0x0000ffffb6b23d08 in __thread_run (arg=0x53ebffe4fe00) at /data/mci/85e522c0f0c3685c4ab1799e0ea99aec/wiredtiger/src/support/thread_group.c:47
#6  0x0000ffffb66c2834 in start_thread () from /lib64/libc.so.6
#7  0x0000ffffb6666e5c [PAC] in thread_start () from /lib64/libc.so.6

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: