format-stress-test-disagg-leader: toolchain installer.sh missing on amazon2023-arm64-nonstandalone

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      format-stress-test-disagg-leader-data-validation-1 on amazon2023-arm64-nonstandalone

      Host: i-069a8a8c283062e19
      Project: wiredtiger
      Commit: 6a0175cf
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/products.sh: line 13: /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/installer.sh: No such file or directory
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      [1781853246:511932][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 278: Cache stuck for too long, giving up: Connection timed out
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      0x5204ffc97000:transaction state dump
      0x5204ffc97000:current ID: 11895434
      0x5204ffc97000:last running ID: 11895434
      0x5204ffc97000:metadata_pinned ID: 11895434
      0x5204ffc97000:oldest ID: 11895434
      0x5204ffc97000:durable timestamp: (0, 31957351)
      0x5204ffc97000:oldest timestamp: (0, 30937802)
      0x5204ffc97000:pinned timestamp: (0, 30937802)
      0x5204ffc97000:stable timestamp: (0, 30937802)
      0x5204ffc97000:stable disaggregated schema epoch: (0, 0)
      0x5204ffc97000:has_durable_timestamp: yes
      0x5204ffc97000:has_oldest_timestamp: yes
      0x5204ffc97000:has_pinned_timestamp: yes
      0x5204ffc97000:has_stable_timestamp: yes
      0x5204ffc97000:has_stable_disaggregated_schema_epoch: no
      0x5204ffc97000:oldest_is_pinned: yes
      0x5204ffc97000:stable_is_pinned: yes
      0x5204ffc97000:checkpoint running: no
      0x5204ffc97000:checkpoint generation: 22
      0x5204ffc97000:checkpoint pinned ID: 0
      0x5204ffc97000:checkpoint txn ID: 0
      0x5204ffc97000:session count: 22
      0x5204ffc97000:Transaction state of active sessions:
      0x5204ffc97000:session ID: 21, txn ID: 0, pinned ID: 11895434, metadata pinned ID: 0, name: WT_CURSOR.search
      0x5204ffc97000:transaction id: 0, mod count: 0, snap min: 11895434, snap max: 11895434, snapshot count: 0, snapshot: [], commit_timestamp: (0, 0), durable_timestamp: (0, 0), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), prepared id: 0, pinned_durable_timestamp: (0, 0), read_timestamp: (0, 30937802), checkpoint LSN: [0,0], full checkpoint: false, flags: 0x00000a04, isolation: WT_ISO_SNAPSHOT, last saved error code: 0, last saved sub-level error code: -32000, last saved error message: last API call was successful
      0x5204ffc97000:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x5204ffc97000:cache dump
      0x5204ffc97000:cache full: no
      0x5204ffc97000:cache clean check: no (27.019%)
      0x5204ffc97000:cache dirty check: yes (20.924%)
      0x5204ffc97000:cache updates check: yes (10.488%)
      0x5204ffc97000:file:T00003.wt_stable(<live>):
      0x5204ffc97000:internal: 519 pages, 36221.37 KB, 5/514 clean/dirty pages, 261.98/35959.39 clean / dirty KB, 119.43 KB max page, 119.43 KB max dirty page
      0x5204ffc97000:leaf: 184502 pages, 319792.38 KB, 83869/100633 clean/dirty pages, 113222.75 /206569.64 /116716.63 clean/dirty/updates KB, 2493.89 KB max page, 2493.89 KB max dirty page
      0x5204ffc97000:file:T00002.wt_stable(<live>):
      0x5204ffc97000:internal: 11 pages, 632.59 KB, 1/10 clean/dirty pages, 82.55/550.05 clean / dirty KB, 95.25 KB max page, 95.25 KB max dirty page
      0x5204ffc97000:leaf: 2870 pages, 229327.13 KB, 130/2740 clean/dirty pages, 21112.37 /208214.76 /89541.18 clean/dirty/updates KB, 2493.89 KB max page, 2493.89 KB max dirty page
      0x5204ffc97000:file:T00001.wt(<live>):
      0x5204ffc97000:internal: 75 pages, 6065.75 KB, 8/67 clean/dirty pages, 2258.38/3807.37 clean / dirty KB, 482.78 KB max page, 177.33 KB max dirty page
      0x5204ffc97000:leaf: 2240 pages, 194925.92 KB, 14/2226 clean/dirty pages, 246.77 /194679.15 /99225.23 clean/dirty/updates KB, 2397.38 KB max page, 2397.38 KB max dirty page
      0x5204ffc97000:file:WiredTigerShared.wt_stable(<live>):
      0x5204ffc97000:internal: 1 pages, 0.85 KB, 1/0 clean/dirty pages, 0.85/0.00 clean / dirty KB, 0.85 KB max page, 0.00 KB max dirty page
      0x5204ffc97000:leaf: 0 pages
      0x5204ffc97000:file:WiredTigerSharedHS.wt_stable(<live>):
      0x5204ffc97000:internal: 1 pages, 0.79 KB, 1/0 clean/dirty pages, 0.79/0.00 clean / dirty KB, 0.79 KB max page, 0.00 KB max dirty page
      0x5204ffc97000:leaf: 0 pages
      0x5204ffc97000:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.24(<live>) eviction disabled at open:
      0x5204ffc97000:internal: 1 pages, 0.48 KB, 1/0 clean/dirty pages, 0.48/0.00 clean / dirty KB, 0.48 KB max page, 0.00 KB max dirty page
      0x5204ffc97000:leaf: 0 pages
      0x5204ffc97000:file:WiredTigerHS.wt(<live>):
      0x5204ffc97000:internal: 1 pages, 0.76 KB, 0/1 clean/dirty pages, 0.00/0.76 clean / dirty KB, 0.76 KB max page, 0.76 KB max dirty page
      0x5204ffc97000:leaf: 0 pages
      0x5204ffc97000:file:WiredTiger.wt(<live>):
      0x5204ffc97000:internal: 1 pages, 0.77 KB, 1/0 clean/dirty pages, 0.77/0.00 clean / dirty KB, 0.77 KB max page, 0.00 KB max dirty page
      0x5204ffc97000:leaf: 1 pages, 14.07 KB, 1/0 clean/dirty pages, 14.07 /0.00 /0.00 clean/dirty/updates KB, 14.07 KB max page, 0.00 KB max dirty page
      0x5204ffc97000:cache dump: total found: 830.02 MB vs tracked inuse 768.54 MB
      0x5204ffc97000:total dirty bytes: 634.55 MB vs tracked dirty 634.55 MB
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      [1781853246:577703][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 795: Error at src/btree/bt_curnext.c:795: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1781853246:577713][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1781853246:577716][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 98: Error at src/evict/evict_thread.c:98: "ret" failed: Connection timed out
      [1781853246:577720][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: eviction thread error: Connection timed out
      [1781853246:577722][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 121: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1781853246:577726][33626:0xffff9098dbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

      format-stress-test-disagg-leader-data-validation-1 task_log

      Logs:

      #0  0x0000ffff94ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff94ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff94a7b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff94a62224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff94e54f38 [PAC] in __wt_abort (session=session@entry=0x5204ffc97000) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff94f005c4 in __wt_panic_func (session=session@entry=0x5204ffc97000, error=error@entry=110, func=func@entry=0xffff95006380 <__PRETTY_FUNCTION__.6> "__evict_thread_run", line=line@entry=121, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0xffff94fa0af0 "eviction thread error") at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/support/err.c:633
      #5  0x0000ffff94e0d328 in __evict_thread_run (session=0x5204ffc97000, thread=0x5204ffe2f1d0) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/evict/evict_thread.c:121
      #6  0x0000ffff94f1c508 in __thread_run (arg=0x5204ffe2f1d0) at /data/mci/0b451b5d33018fa3f1ae6702d17741c1/wiredtiger/src/support/thread_group.c:32
      #7  0x0000ffff94ac2834 in start_thread () from /lib64/libc.so.6
      #8  0x0000ffff94a66e5c [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: