DUPLICATE cache stuck, bt_curnext not found

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Test Format

      format-stress-test-no-barrier on amazon2023-stress-tests-armv9

      Host: i-04cb0c1215e17b2d2
      Project: wiredtiger
      Commit: f53c3259
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-no-barrier task_log

      Logs:

          [1765883056:114326][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
      
      

      logs

      format-stress-test-no-barrier task_log

      Logs:

          0x7203ffe946e8:transaction state dump
          0x7203ffe946e8:current ID: 2779
          0x7203ffe946e8:last running ID: 2779
          0x7203ffe946e8:metadata_pinned ID: 2778
          0x7203ffe946e8:oldest ID: 2779
          0x7203ffe946e8:durable timestamp: (0, 10376)
          0x7203ffe946e8:oldest timestamp: (0, 8040)
          0x7203ffe946e8:pinned timestamp: (0, 8040)
          0x7203ffe946e8:stable timestamp: (0, 8040)
          0x7203ffe946e8:has_durable_timestamp: yes
          0x7203ffe946e8:has_oldest_timestamp: yes
          0x7203ffe946e8:has_pinned_timestamp: yes
          0x7203ffe946e8:has_stable_timestamp: yes
          0x7203ffe946e8:oldest_is_pinned: yes
          0x7203ffe946e8:stable_is_pinned: yes
          0x7203ffe946e8:checkpoint running: yes
          0x7203ffe946e8:checkpoint generation: 14
          0x7203ffe946e8:checkpoint pinned ID: 2778
          0x7203ffe946e8:checkpoint txn ID: 2778
          0x7203ffe946e8:session count: 42
          0x7203ffe946e8:Transaction state of active sessions:
          0x7203ffe946e8:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
          0x7203ffe946e8:cache dump
          0x7203ffe946e8:cache full: no
          0x7203ffe946e8:cache clean check: no (33.147%)
          0x7203ffe946e8:cache dirty check: yes (20.136%)
          0x7203ffe946e8:cache updates check: no (1.358%)
          0x7203ffe946e8:file:F00002(<live>):
          0x7203ffe946e8:internal: 37 pages, 50638.67 KB, 37/0 clean/dirty pages, 50638.67/0.00 clean / dirty KB, 1486.54 KB max page, 0.00 KB max dirty page
          0x7203ffe946e8:leaf: 11085 pages, 23505.40 KB, 9605/1480 clean/dirty pages, 19591.31 /3914.09 /1272.00 clean/dirty/updates KB, 82.80 KB max page, 82.75 KB max dirty page
          0x7203ffe946e8:file:T00001.wt(<live>):
          0x7203ffe946e8:internal: 1 pages, 533.62 KB, 1/0 clean/dirty pages, 533.62/0.00 clean / dirty KB, 533.62 KB max page, 0.00 KB max dirty page
          0x7203ffe946e8:leaf: 2780 pages, 274940.42 KB, 1834/946 clean/dirty pages, 174397.43 /100542.99 /11394.22 clean/dirty/updates KB, 197.68 KB max page, 197.68 KB max dirty page
          0x7203ffe946e8:file:T00003.wt(<live>):
          0x7203ffe946e8:internal: 9 pages, 220.18 KB, 2/7 clean/dirty pages, 56.55/163.63 clean / dirty KB, 28.74 KB max page, 27.81 KB max dirty page
          0x7203ffe946e8:leaf: 1083 pages, 565329.84 KB, 227/856 clean/dirty pages, 113841.15 /451488.69 /24807.35 clean/dirty/updates KB, 634.37 KB max page, 634.37 KB max dirty page
          0x7203ffe946e8:file:WiredTigerHS.wt(<live>) eviction disabled at open:
          0x7203ffe946e8:internal: 1 pages, 0.41 KB, 1/0 clean/dirty pages, 0.41/0.00 clean / dirty KB, 0.41 KB max page, 0.00 KB max dirty page
          0x7203ffe946e8:leaf: 1 pages, 0.23 KB, 1/0 clean/dirty pages, 0.23 /0.00 /0.00 clean/dirty/updates KB, 0.23 KB max page, 0.00 KB max dirty page
          0x7203ffe946e8:file:WiredTiger.wt(<live>):
          0x7203ffe946e8:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
          0x7203ffe946e8:leaf: 1 pages, 24.21 KB, 0/1 clean/dirty pages, 0.00 /24.21 /17.13 clean/dirty/updates KB, 24.21 KB max page, 24.21 KB max dirty page
          0x7203ffe946e8:cache dump: total found: 965.24 MB vs tracked inuse 893.74 MB
          0x7203ffe946e8:total dirty bytes: 543.10 MB vs tracked dirty 543.10 MB
      

      logs

      format-stress-test-no-barrier task_log

      Logs:

          [1765883056:120409][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 797: Error at src/btree/bt_curnext.c:797: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
          [1765883056:120419][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
          [1765883056:120422][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
          [1765883056:120427][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
          [1765883056:120429][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
          [1765883056:120433][112144:0xffff971bdbc0], t, file:WiredTigerHS.wt, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

      format-stress-test-no-barrier task_log

      Logs:

          /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/cmake_build/test/format/../../libwiredtiger.so.12.0.0(__wt_panic_func+0x194)[0xffff9a2cea84]
      
      

      logs

      format-stress-test-no-barrier task_log

      Logs:

      test/format run configuration highlights
      

      logs

      format-stress-test-no-barrier task_log

      Logs:

      #0  0x0000ffff99ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff99ebf7b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff99e763a0 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff99e62264 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9a22cf9c [PAC] in __wt_abort (session=session@entry=0x7203ffe946e8) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff9a2cea84 in __wt_panic_func (session=session@entry=0x7203ffe946e8, error=error@entry=110, func=func@entry=0xffff9a3c1140 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff9a361338 "eviction thread error") at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/support/err.c:611
      #5  0x0000ffff9a1e3190 in __evict_thread_run (session=0x7203ffe946e8, thread=0x7203ffc00a50) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/evict/evict_lru.c:359
      #6  0x0000ffff9a2e9764 in __thread_run (arg=0x7203ffc00a50) at /data/mci/9d8b909f53df1bb9858b716657ac788a/wiredtiger/src/support/thread_group.c:32
      #7  0x0000ffff99ebdb78 in start_thread () from /lib64/libc.so.6
      #8  0x0000ffff99f2acdc [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: