failed: format-stress-test-2 on amazon2023-stress-tests-armv9 [wiredtiger @ 6004b445]

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction

      format-stress-test-2 on amazon2023-stress-tests-armv9

      Host: i-00914d9d4d77c2eec
      Project: wiredtiger
      Commit: 6004b445
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-2 task_log

      Logs:

              echo "Clone failed after $max_attempts attempts."
      
      

      logs

      format-stress-test-2 task_log

      Logs:

          [1764988339:486437][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
      
      

      logs

      format-stress-test-2 task_log

      Logs:

          0x3066bfe946e8:transaction state dump
          0x3066bfe946e8:current ID: 1192
          0x3066bfe946e8:last running ID: 1192
          0x3066bfe946e8:metadata_pinned ID: 1191
          0x3066bfe946e8:oldest ID: 1192
          0x3066bfe946e8:durable timestamp: (0, 6092)
          0x3066bfe946e8:oldest timestamp: (0, 5160)
          0x3066bfe946e8:pinned timestamp: (0, 5160)
          0x3066bfe946e8:stable timestamp: (0, 5160)
          0x3066bfe946e8:has_durable_timestamp: yes
          0x3066bfe946e8:has_oldest_timestamp: yes
          0x3066bfe946e8:has_pinned_timestamp: yes
          0x3066bfe946e8:has_stable_timestamp: yes
          0x3066bfe946e8:oldest_is_pinned: yes
          0x3066bfe946e8:stable_is_pinned: yes
          0x3066bfe946e8:checkpoint running: yes
          0x3066bfe946e8:checkpoint generation: 35
          0x3066bfe946e8:checkpoint pinned ID: 1191
          0x3066bfe946e8:checkpoint txn ID: 1191
          0x3066bfe946e8:session count: 48
          0x3066bfe946e8:Transaction state of active sessions:
          0x3066bfe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
          0x3066bfe946e8:cache dump
          0x3066bfe946e8:cache full: no
          0x3066bfe946e8:cache clean check: no (29.845%)
          0x3066bfe946e8:cache dirty check: yes (20.477%)
          0x3066bfe946e8:cache updates check: no (1.689%)
          0x3066bfe946e8:file:F00002(<live>):
          0x3066bfe946e8:internal: 5 pages, 97.38 KB, 5/0 clean/dirty pages, 97.38/0.00 clean / dirty KB, 28.43 KB max page, 0.00 KB max dirty page
          0x3066bfe946e8:leaf: 506 pages, 260019.11 KB, 141/365 clean/dirty pages, 69663.63 /190355.48 /13002.43 clean/dirty/updates KB, 622.00 KB max page, 622.00 KB max dirty page
          0x3066bfe946e8:file:T00001.wt(<live>):
          0x3066bfe946e8:internal: 137 pages, 941.04 KB, 137/0 clean/dirty pages, 941.04/0.00 clean / dirty KB, 8.16 KB max page, 0.00 KB max dirty page
          0x3066bfe946e8:leaf: 2646 pages, 117373.86 KB, 2053/593 clean/dirty pages, 88686.39 /28687.47 /2981.47 clean/dirty/updates KB, 226.43 KB max page, 226.43 KB max dirty page
          0x3066bfe946e8:file:T00003.wt(<live>):
          0x3066bfe946e8:internal: 1 pages, 66.95 KB, 0/1 clean/dirty pages, 0.00/66.95 clean / dirty KB, 66.95 KB max page, 66.95 KB max dirty page
          0x3066bfe946e8:leaf: 350 pages, 201023.22 KB, 42/308 clean/dirty pages, 22462.84 /178560.38 /16805.46 clean/dirty/updates KB, 684.48 KB max page, 684.48 KB max dirty page
          0x3066bfe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
          0x3066bfe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
          0x3066bfe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
          0x3066bfe946e8:file:WiredTiger.wt(<live>):
          0x3066bfe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
          0x3066bfe946e8:leaf: 1 pages, 17.93 KB, 0/1 clean/dirty pages, 0.00 /17.93 /10.94 clean/dirty/updates KB, 17.93 KB max page, 17.93 KB max dirty page
          0x3066bfe946e8:cache dump: total found: 611.23 MB vs tracked inuse 565.96 MB
          0x3066bfe946e8:total dirty bytes: 388.37 MB vs tracked dirty 388.37 MB
      

      logs

      format-stress-test-2 task_log

      Logs:

          [1764988339:487772][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 797: Error at src/btree/bt_curnext.c:797: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
          [1764988339:487777][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
          [1764988339:487780][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
          [1764988339:487783][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
          [1764988339:487785][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
          [1764988339:487788][152705:0xffff879fdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

      format-stress-test-2 task_log

      Logs:

          /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff882cd1d4]
      
      

      logs

      format-stress-test-2 task_log

      Logs:

      test/format run configuration highlights
      

      logs

      format-stress-test-2 task_log

      Logs:

      #0  0x0000ffff87ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff87ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff87e763a0 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff87e62264 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff8822b9c4 [PAC] in __wt_abort (session=session@entry=0x3066bfe946e8) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff882cd1d4 in __wt_panic_func (session=session@entry=0x3066bfe946e8, error=error@entry=110, func=func@entry=0xffff883be318 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff8835e840 "eviction thread error") at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/support/err.c:611
      #5  0x0000ffff881e1bf8 in __evict_thread_run (session=0x3066bfe946e8, thread=0x3066bfc00870) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/evict/evict_lru.c:359
      #6  0x0000ffff882e7ddc in __thread_run (arg=0x3066bfc00870) at /data/mci/8528dbb0b0481e1055629852b30274c5/wiredtiger/src/support/thread_group.c:32
      #7  0x0000ffff87ebdb78 in start_thread () from /lib64/libc.so.6
      #8  0x0000ffff87f2acdc [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: