failed: format-stress-test-disagg-switch-data-validation-1 on amazon2023-disagg-stress [wiredtiger @ 4a5646cb]

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction

      format-stress-test-disagg-switch-data-validation-1 on amazon2023-disagg-stress

      Host: i-094ad963de348cda6
      Project: wiredtiger
      Commit: 4a5646cb
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      [1775528674:308848][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      0x3253ef8b3f60:transaction state dump
0x3253ef8b3f60:current ID: 320726
0x3253ef8b3f60:last running ID: 320726
0x3253ef8b3f60:metadata_pinned ID: 320726
0x3253ef8b3f60:oldest ID: 320726
0x3253ef8b3f60:durable timestamp: (0, 3849108)
0x3253ef8b3f60:oldest timestamp: (0, 3848910)
0x3253ef8b3f60:pinned timestamp: (0, 3848910)
0x3253ef8b3f60:stable timestamp: (0, 3848997)
0x3253ef8b3f60:has_durable_timestamp: yes
0x3253ef8b3f60:has_oldest_timestamp: yes
0x3253ef8b3f60:has_pinned_timestamp: yes
0x3253ef8b3f60:has_stable_timestamp: yes
0x3253ef8b3f60:oldest_is_pinned: yes
0x3253ef8b3f60:stable_is_pinned: no
0x3253ef8b3f60:checkpoint running: no
0x3253ef8b3f60:checkpoint generation: 1
0x3253ef8b3f60:checkpoint pinned ID: 0
0x3253ef8b3f60:checkpoint txn ID: 0
0x3253ef8b3f60:session count: 39
0x3253ef8b3f60:Transaction state of active sessions:
0x3253ef8b3f60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
0x3253ef8b3f60:cache dump
0x3253ef8b3f60:cache full: no
0x3253ef8b3f60:cache clean check: no (33.442%)
0x3253ef8b3f60:cache dirty check: no (13.674%)
0x3253ef8b3f60:cache updates check: yes (13.234%)
0x3253ef8b3f60:file:T00003.wt_stable(<live>):
0x3253ef8b3f60:internal: 11 pages, 1550.17 KB, 1/10 clean/dirty pages, 722.20/827.97 clean / dirty KB, 722.20 KB max page, 180.16 KB max dirty page
0x3253ef8b3f60:leaf: 3695 pages, 196770.49 KB, 3543/152 clean/dirty pages, 189121.75 /7648.75 /467.85 clean/dirty/updates KB, 206.74 KB max page, 72.83 KB max dirty page
0x3253ef8b3f60:file:T00002.wt_stable(<live>):
0x3253ef8b3f60:internal: 3901 pages, 29524.67 KB, 135/3766 clean/dirty pages, 1867.62/27657.05 clean / dirty KB, 242.40 KB max page, 83.74 KB max dirty page
0x3253ef8b3f60:leaf: 74708 pages, 170978.75 KB, 74550/158 clean/dirty pages, 170562.47 /416.28 /110.46 clean/dirty/updates KB, 180.63 KB max page, 7.32 KB max dirty page
0x3253ef8b3f60:file:T00003.wt_ingest(<live>):
0x3253ef8b3f60:internal: 1 pages, 11.56 KB, 0/1 clean/dirty pages, 0.00/11.56 clean / dirty KB, 11.56 KB max page, 11.56 KB max dirty page
0x3253ef8b3f60:leaf: 58 pages, 177033.63 KB, 0/58 clean/dirty pages, 0.00 /177033.63 /174520.76 clean/dirty/updates KB, 6811.61 KB max page, 6811.61 KB max dirty page
0x3253ef8b3f60:file:T00002.wt_ingest(<live>):
0x3253ef8b3f60:internal: 1 pages, 181.35 KB, 0/1 clean/dirty pages, 0.00/181.35 clean / dirty KB, 181.35 KB max page, 181.35 KB max dirty page
0x3253ef8b3f60:leaf: 740 pages, 212110.87 KB, 0/740 clean/dirty pages, 0.00 /212110.87 /210227.29 clean/dirty/updates KB, 1846.52 KB max page, 1846.52 KB max dirty page
0x3253ef8b3f60:file:T00001.wt(<live>):
0x3253ef8b3f60:internal: 335 pages, 22791.18 KB, 30/305 clean/dirty pages, 5596.76/17194.42 clean / dirty KB, 562.44 KB max page, 160.95 KB max dirty page
0x3253ef8b3f60:leaf: 31181 pages, 163021.72 KB, 31023/158 clean/dirty pages, 161990.44 /1031.27 /138.19 clean/dirty/updates KB, 240.50 KB max page, 10.09 KB max dirty page
0x3253ef8b3f60:file:WiredTigerSharedHS.wt_stable(<live>):
0x3253ef8b3f60:internal: 1 pages, 0.79 KB, 0/1 clean/dirty pages, 0.00/0.79 clean / dirty KB, 0.79 KB max page, 0.79 KB max dirty page
0x3253ef8b3f60:leaf: 1 pages, 27.04 KB, 1/0 clean/dirty pages, 27.04 /0.00 /0.00 clean/dirty/updates KB, 27.04 KB max page, 0.00 KB max dirty page
0x3253ef8b3f60:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.6(<live>) eviction disabled at open:
0x3253ef8b3f60:internal: 1 pages, 0.47 KB, 1/0 clean/dirty pages, 0.47/0.00 clean / dirty KB, 0.47 KB max page, 0.00 KB max dirty page
0x3253ef8b3f60:leaf: 0 pages
0x3253ef8b3f60:file:WiredTigerHS.wt(<live>):
0x3253ef8b3f60:internal: 1 pages, 0.74 KB, 0/1 clean/dirty pages, 0.00/0.74 clean / dirty KB, 0.74 KB max page, 0.74 KB max dirty page
0x3253ef8b3f60:leaf: 0 pages
0x3253ef8b3f60:file:WiredTiger.wt(<live>):
0x3253ef8b3f60:internal: 1 pages, 0.77 KB, 0/1 clean/dirty pages, 0.00/0.77 clean / dirty KB, 0.77 KB max page, 0.77 KB max dirty page
0x3253ef8b3f60:leaf: 1 pages, 13.79 KB, 1/0 clean/dirty pages, 13.79 /0.00 /0.00 clean/dirty/updates KB, 13.79 KB max page, 0.00 KB max dirty page
0x3253ef8b3f60:cache dump: total found: 1027.28 MB vs tracked inuse 951.23 MB
0x3253ef8b3f60:total dirty bytes: 433.71 MB vs tracked dirty 433.75 MB

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      [1775528674:343369][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 795: Error at src/btree/bt_curnext.c:795: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
[1775528674:343380][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
[1775528674:343383][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
[1775528674:343387][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
[1775528674:343389][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1775528674:343392][3755:0xffff8affdbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      #0  0x0000ffff8b8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#0  0x0000ffff8b8c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x0000ffff8b87b320 [PAC] in raise () from /lib64/libc.so.6
#2  0x0000ffff8b862224 [PAC] in abort () from /lib64/libc.so.6
#3  0x0000ffff8bc3cc04 [PAC] in __wt_abort (session=session@entry=0x3253ef8b3f60) at /data/mci/7d5e653b84faff4f9b0fb790d3bf8524/wiredtiger/src/os_common/os_abort.c:32
#4  0x0000ffff8bce2064 in __wt_panic_func (session=session@entry=0x3253ef8b3f60, error=error@entry=110, func=func@entry=0xffff8bdd9970 <__PRETTY_FUNCTION__.30> "__evict_thread_run", line=line@entry=359, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff8bd789d0 "eviction thread error") at /data/mci/7d5e653b84faff4f9b0fb790d3bf8524/wiredtiger/src/support/err.c:611
#5  0x0000ffff8bbf2080 in __evict_thread_run (session=0x3253ef8b3f60, thread=0x3253ffe067d0) at /data/mci/7d5e653b84faff4f9b0fb790d3bf8524/wiredtiger/src/evict/evict_lru.c:359
#6  0x0000ffff8bcfd650 in __thread_run (arg=0x3253ffe067d0) at /data/mci/7d5e653b84faff4f9b0fb790d3bf8524/wiredtiger/src/support/thread_group.c:32
#7  0x0000ffff8b8c2834 in start_thread () from /lib64/libc.so.6
#8  0x0000ffff8b866e5c [PAC] in thread_start () from /lib64/libc.so.6

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: