Disagg drain worker panics when the ingest table's stable pair is gone from the metadata

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      unit-test-bucket02 on ubuntu2004-nonstandalone

      Host: i-0e9d5d2ddc76d4d76
      Project: wiredtiger
      Commit: 4e3b4599
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      unit-test-bucket02 task_log

      Logs:

      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x00007f783fbbb859 in __GI_abort () at abort.c:79
      #2  0x00007f783f1c9ebe in __wt_abort (session=session@entry=0x72bcbd2aa000) at /data/mci/4c24c4d10326c5eaaf6213b50f55a9b8/wiredtiger/src/os_common/os_abort.c:32
      #3  0x00007f783f280850 in __wt_panic_func (session=session@entry=0x72bcbd2aa000, error=error@entry=2, func=func@entry=0x7f783f3a4718 <__PRETTY_FUNCTION__.7> "__thread_run", line=line@entry=47, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0x7f783f365640 "Unrecoverable utility thread error") at /data/mci/4c24c4d10326c5eaaf6213b50f55a9b8/wiredtiger/src/support/err.c:633
      #4  0x00007f783f29e82e in __thread_run (arg=0x72bcbff98cd0) at /data/mci/4c24c4d10326c5eaaf6213b50f55a9b8/wiredtiger/src/support/thread_group.c:47
      #5  0x00007f783feef609 in start_thread (arg=<optimized out>) at pthread_create.c:477
      #6  0x00007f783fcb8353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      logs

      unit-test-bucket02 task_log

      Logs:

      [1788915523:362182][14517:0x7f7827f33700], disagg-drain: [WT_VERB_DEFAULT][ERROR]: __layered_drain_worker_run, 859: Failed to drain ingest and truncate list for "file:test_layered_schema33_on_follower.wt_ingest": No such file or directory
      [1788915523:362326][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 326: Error at src/cursor/cur_file.c:326: "ret" failed: WT_NOTFOUND: item not found
      [1788915523:362351][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
      [1788915523:362367][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_dhandle_config_set, 64: Error at src/conn/conn_dhandle.c:64: "ret" failed: No such file or directory
      [1788915523:362383][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_open, 651: Error at src/conn/conn_dhandle.c:651: "__conn_dhandle_config_set(session)" failed: No such file or directory
      [1788915523:362407][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_session_get_dhandle, 1102: Error at src/session/session_dhandle.c:1102: "ret" failed: No such file or directory
      [1788915523:362423][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_search, 326: Error at src/cursor/cur_file.c:326: "ret" failed: WT_NOTFOUND: item not found
      [1788915523:362437][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_metadata_search, 341: Error at src/meta/meta_table.c:341: "ret" failed: WT_NOTFOUND: item not found
      [1788915523:362449][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __conn_dhandle_config_set, 64: Error at src/conn/conn_dhandle.c:64: "ret" failed: No such file or directory
      [1788915523:362460][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_conn_dhandle_open, 651: Error at src/conn/conn_dhandle.c:651: "__conn_dhandle_config_set(session)" failed: No such file or directory
      [1788915523:362472][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_session_get_dhandle, 1102: Error at src/session/session_dhandle.c:1102: "ret" failed: No such file or directory
      [1788915523:362484][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_curfile_open, 1347: Error at src/cursor/cur_file.c:1347: "ret" failed: No such file or directory
      [1788915523:362496][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __session_open_cursor_int, 696: Error at src/session/session_api.c:696: "__wt_curfile_open(session, uri, owner, cfg, cursorp)" failed: No such file or directory
      [1788915523:362509][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_open_cursor, 805: Error at src/session/session_api.c:805: "__session_open_cursor_int(session, uri, owner, NULL, cfg, hash_value, cursorp)" failed: No such file or directory
      [1788915523:362523][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __layered_copy_ingest_table, 466: Error at src/conn/conn_layered_ingest.c:466: "__wt_open_cursor(session, stable_uri_buf->data, NULL, open_cfg, &stable_cursor)" failed: No such file or directory
      [1788915523:362539][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __layered_drain_ingest_table_and_truncate_list, 798: Error at src/conn/conn_layered_ingest.c:798: "__layered_copy_ingest_table(session, ingest_uri, WT_TS_NONE, WT_TS_MAX)" failed: No such file or directory
      [1788915523:362553][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __layered_drain_worker_run, 859: Error at src/conn/conn_layered_ingest.c:859: "ret" failed: No such file or directory
      [1788915523:362565][14517:0x7f7827f33700], disagg-drain: [WT_VERB_ERROR_RETURNS][ERROR]: __thread_run, 32: Error at src/support/thread_group.c:32: "thread->run_func(session, thread)" failed: No such file or directory
      [1788915523:362578][14517:0x7f7827f33700], disagg-drain: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: Unrecoverable utility thread error: No such file or directory
      [1788915523:362598][14517:0x7f7827f33700], disagg-drain: [WT_VERB_DEFAULT][ERROR]: __thread_run, 47: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1788915523:362612][14517:0x7f7827f33700], disagg-drain: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

            Assignee:
            Chenhao Qu
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: