test/format (disagg.mode=switch) cache stuck

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-test-disagg-switch-5 on amazon2023-disagg-stress

      Host: i-0712165f9cb84cdcb
      Project: wiredtiger
      Commit: 6d5113ee
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-5 task_log

      Logs:

      [1757557009:258623][224644:0xffff211beb80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 541: Cache stuck for too long, giving up: Connection timed out
      
      

      logs

      format-stress-test-disagg-switch-5 task_log

      Logs:

      0x7124bfc7a880:transaction state dump
      0x7124bfc7a880:current ID: 1221272
      0x7124bfc7a880:last running ID: 1221272
      0x7124bfc7a880:metadata_pinned ID: 1221272
      0x7124bfc7a880:oldest ID: 1221272
      0x7124bfc7a880:durable timestamp: (0, 1439747)
      0x7124bfc7a880:oldest timestamp: (0, 1091720)
      0x7124bfc7a880:pinned timestamp: (0, 1091720)
      0x7124bfc7a880:stable timestamp: (0, 1439591)
      0x7124bfc7a880:has_durable_timestamp: yes
      0x7124bfc7a880:has_oldest_timestamp: yes
      0x7124bfc7a880:has_pinned_timestamp: yes
      0x7124bfc7a880:has_stable_timestamp: yes
      0x7124bfc7a880:oldest_is_pinned: yes
      0x7124bfc7a880:stable_is_pinned: no
      0x7124bfc7a880:checkpoint running: no
      0x7124bfc7a880:checkpoint generation: 4
      0x7124bfc7a880:checkpoint pinned ID: 0
      0x7124bfc7a880:checkpoint txn ID: 0
      0x7124bfc7a880:session count: 26
      0x7124bfc7a880:Transaction state of active sessions:
      0x7124bfc7a880:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7124bfc7a880:cache dump
      0x7124bfc7a880:cache full: no
      0x7124bfc7a880:cache clean check: no (21.417%)
      0x7124bfc7a880:cache dirty check: yes (20.001%)
      0x7124bfc7a880:cache updates check: no (6.892%)
      0x7124bfc7a880:file:T00003.wt_ingest(<live>):
      0x7124bfc7a880:internal: 584 pages, 18337.84 KB, 0/584 clean/dirty pages, 0.00/18337.84 clean / dirty KB, 1974.76 KB max page, 1974.76 KB max dirty page
      0x7124bfc7a880:leaf: 85651 pages, 284943.65 KB, 0/85651 clean/dirty pages, 0.00 /284943.65 /72794.57 clean/dirty/updates KB, 2002.06 KB max page, 2002.06 KB max dirty page
      0x7124bfc7a880:file:T00002.wt_ingest(<live>):
      0x7124bfc7a880:internal: 102 pages, 2813.38 KB, 0/102 clean/dirty pages, 0.00/2813.38 clean / dirty KB, 194.40 KB max page, 194.40 KB max dirty page
      0x7124bfc7a880:leaf: 12044 pages, 203570.31 KB, 0/12044 clean/dirty pages, 0.00 /203570.31 /60964.88 clean/dirty/updates KB, 1050.61 KB max page, 1050.61 KB max dirty page
      0x7124bfc7a880:file:T00001.wt_stable(<live>) eviction disabled at open:
      0x7124bfc7a880:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page
      0x7124bfc7a880:leaf: 0 pages
      0x7124bfc7a880:file:T00001.wt_ingest(<live>):
      0x7124bfc7a880:internal: 549 pages, 33701.33 KB, 0/549 clean/dirty pages, 0.00/33701.33 clean / dirty KB, 1738.98 KB max page, 1738.98 KB max dirty page
      0x7124bfc7a880:leaf: 63321 pages, 286342.25 KB, 0/63321 clean/dirty pages, 0.00 /286342.25 /133219.57 clean/dirty/updates KB, 1343.30 KB max page, 1343.30 KB max dirty page
      0x7124bfc7a880:file:WiredTigerSharedHS.wt_stable(<live>) eviction disabled at open:
      0x7124bfc7a880:internal: 1 pages, 0.50 KB, 1/0 clean/dirty pages, 0.50/0.00 clean / dirty KB, 0.50 KB max page, 0.00 KB max dirty page
      0x7124bfc7a880:leaf: 0 pages
      0x7124bfc7a880:file:WiredTigerHS.wt(<live>) eviction disabled at open:
      0x7124bfc7a880:internal: 1 pages, 0.40 KB, 1/0 clean/dirty pages, 0.40/0.00 clean / dirty KB, 0.40 KB max page, 0.00 KB max dirty page
      0x7124bfc7a880:leaf: 0 pages
      0x7124bfc7a880:file:WiredTiger.wt(<live>):
      0x7124bfc7a880:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
      0x7124bfc7a880:leaf: 1 pages, 17.31 KB, 1/0 clean/dirty pages, 17.31 /0.00 /7.70 clean/dirty/updates KB, 17.31 KB max page, 0.00 KB max dirty page
      0x7124bfc7a880:cache dump: total found: 875.10 MB vs tracked inuse 810.28 MB
      0x7124bfc7a880:total dirty bytes: 810.26 MB vs tracked dirty 810.26 MB
      

      logs

      format-stress-test-disagg-switch-5 task_log

      Logs:

      [1757557009:311281][224644:0xffff211beb80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 358: eviction thread error: Connection timed out
      [1757557009:311291][224644:0xffff211beb80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 358: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1757557009:311295][224644:0xffff211beb80], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

      format-stress-test-disagg-switch-5 task_log

      Logs:

      #0  0x0000ffffa440e9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffffa440e9b4 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffffa43c53a0 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffffa43b1264 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffffa41c7738 [PAC] in __wt_abort (session=session@entry=0x7124bfc7a880) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/os_common/os_abort.c:31
      #4  0x0000ffffa424dbac in __wt_panic_func (session=session@entry=0x7124bfc7a880, error=error@entry=110, func=func@entry=0xffffa42d8d78 <__PRETTY_FUNCTION__.21> "__evict_thread_run", line=line@entry=358, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffffa42a3850 "eviction thread error") at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/support/err.c:572
      #5  0x0000ffffa418f6b0 in __evict_thread_run (session=0x7124bfc7a880, thread=0x7124bfe08d70) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/evict/evict_lru.c:358
      #6  0x0000ffffa4266398 in __thread_run (arg=0x7124bfe08d70) at /data/mci/bef9bd7a4628d4b91c56d038ea547c61/wiredtiger/src/support/thread_group.c:31
      #7  0x0000ffffa440cd78 in start_thread () from /lib64/libc.so.6
      #8  0x0000ffffa4479ddc [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: