test/format cache stuck panic over updates trigger

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-test-disagg-leader-data-validation-2 on amazon2023-stress-tests-arm64

      Host: i-0a2291d218b071ee6
      Project: wiredtiger
      Commit: 92445b64
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

          [1785511961:836153][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 278: Cache stuck for too long, giving up: Connection timed out
      

      logs

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

          0x509a7fc92920:cache full: no
          0x509a7fc92920:cache clean check: no (28.659%)
          0x509a7fc92920:cache dirty check: no (19.165%)
          0x509a7fc92920:cache updates check: yes (10.674%)
      

      logs

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

          0x509a7fc92920:cache dump: total found: 880.28 MB vs tracked inuse 815.09 MB
          0x509a7fc92920:total dirty bytes: 597.44 MB vs tracked dirty 598.11 MB
      

      logs

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

          [1785511961:865414][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 803: Error at src/btree/bt_curnext.c:803: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
          [1785511961:865424][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
          [1785511961:865429][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 98: Error at src/evict/evict_thread.c:98: "ret" failed: Connection timed out
          [1785511961:865433][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: Connection timed out
          [1785511961:865436][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: the process must exit and restart: WT_PANIC: WiredTiger library panic
          [1785511961:865439][15427:0xffff7dfddbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

      test/format run configuration highlights
      

      logs

      format-stress-test-disagg-leader-data-validation-2 task_log

      Logs:

      #0  0x0000ffff7f8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff7f8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff7f87b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff7f862224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff7fc5f1cc [PAC] in __wt_abort (session=session@entry=0x509a7fc92920) at /data/mci/b117b78e4bfda6c1337412bdde6199fe/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff7fd0bdec in __wt_panic_func (session=session@entry=0x509a7fc92920, error=error@entry=110, func=func@entry=0xffff7fe17b68 <__PRETTY_FUNCTION__.6> "__evict_thread_run", line=line@entry=121, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff7fdb0b28 "eviction thread error") at /data/mci/b117b78e4bfda6c1337412bdde6199fe/wiredtiger/src/support/err.c:633
      #5  0x0000ffff7fc16e84 in __evict_thread_run (session=0x509a7fc92920, thread=0x509a7fe45680) at /data/mci/b117b78e4bfda6c1337412bdde6199fe/wiredtiger/src/evict/evict_thread.c:121
      #6  0x0000ffff7fd28794 in __thread_run (arg=0x509a7fe45680) at /data/mci/b117b78e4bfda6c1337412bdde6199fe/wiredtiger/src/support/thread_group.c:32
      #7  0x0000ffff7f8c2834 in start_thread () from /lib64/libc.so.6
      #8  0x0000ffff7f866e5c [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: