failed: schema-disagg-abort-follower-stepup-test-disagg-1 on amazon2023-disagg-asan-stress [wiredtiger-disagg @ 5cc20887]

XMLWordPrintableJSON

      schema-disagg-abort-follower-stepup-test-disagg-1 on amazon2023-disagg-asan-stress

      Host: i-0c53eb3e82edf2394
      Project: wiredtiger-disagg
      Commit: 5cc20887
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      schema-disagg-abort-follower-stepup-test-disagg-1 task_log

      Logs:

      Task logger initialized (agent version '2026-08-26' from Evergreen build revision 'a74e5e27fc4aa76b91a1c485a1613ca06d036c2f').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_follower_stepup_test_disagg_1_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 17.367989ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 50.92µs.
      Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
      mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 not installed, downloading...
      Autodetected Amazon Linux 2023 (OS_ID: amazon2023) on aarch64
      Autodetected DOWNLOAD_URL: https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/latest
      Overriding REVISION_ID, will download toolchain for 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542
      Ensuring /opt/mongodbtoolchain/revisions exists and has correct ownership...
      Downloading toolchain to /data/mci/81813e47d5e7c6e92e7d5e1409ce8476/tmp/tmp.TuI2FfeiXF/mongodbtoolchain.tar.gz from https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/mongodbtoolchain-93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542.tar.gz
      Getting toolchain revision from archive...
      Checking downloaded archive integrity...
      Extracting toolchain revision 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 from archive...
      Running post install script...
      Existing v2 found, uninstalling...
      Existing v3 found, uninstalling...
      Existing v4 found, uninstalling...
      Existing v5 found, uninstalling...
      Installing v2...
      Installing v3...
      Installing v4...
      Installing v5...
      Cleaning up...
      Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3m0.061002878s.
      Finished running pre-task commands in 3m0.078720726s.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 8.081947364s.
      Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
        [ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
        export TESTUTIL_TSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 0 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3631
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_follower_stepup_test_disagg_1_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-128-99-145.ec2.internal
      INVOCATION_ID=21549a27a0a94a84999161d8f1baa98a
      JASPER_ID=0c3f784c-489a-46c6-9e2c-5fddab97520e
      JASPER_MANAGER=cac0d352-e09d-4b44-a767-9e54440d16fb
      JOURNAL_STREAM=8:23136
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      LOGNAME=ec2-user
      LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/test/evergreen/asan_leaks.supp
      MAIL=/var/spool/mail/ec2-user
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      PWD=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3467
      S_COLORS=auto
      TEMP=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/tmp
      TESTUTIL_BYPASS_ASAN=1
      TMP=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/tmp
      TMPDIR=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      echo "Dump Environment"
      printenv | sort
      build_dir=$(cd ../../../ && pwd)
      for i in $(seq 10); do
        echo "Iteration $i/10"
        ./test_schema_disagg_abort -b "$build_dir" -r f -s 30 -T 12 -u 64 -t 300
      Iteration 1/10
      Running test command: ./test_schema_disagg_abort -b /data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build -r f -s 30 -T 12 -u 64 -t 300
      Parent: roles f; 12 schema threads; pool 64 slots; switch every 30s; kill leader@0 follower@0 lone@0; stop at 300s
      CONFIG: test_schema_disagg_abort -r f -s 30 -u 64 -T 12 -t 300 -PSD5851103,E12632315
      Running test command: /data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A f -h WT_TEST.test_schema_disagg_abort -b /data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build -T 12 -u 64 -s 30 -PSD5851103,E12632315
      Node 0: starting as follower
      Parent: directing switch 1
      Node 0: now leader
      Node 0: switch 1 complete
      Node 0: checkpoint 1 complete; stable timestamp = 911
      Node 0: checkpoint 2 complete; stable timestamp = 960
      Node 0: checkpoint 3 complete; stable timestamp = 1033
      Node 0: checkpoint 4 complete; stable timestamp = 1192
      Node 0: checkpoint 5 complete; stable timestamp = 1375
      Node 0: checkpoint 6 complete; stable timestamp = 1551
      Node 0: checkpoint 7 complete; stable timestamp = 1753
      Node 0: checkpoint 8 complete; stable timestamp = 1822
      Parent: directing switch 2
      Node 0: checkpoint 9 complete; stable timestamp = 2042
      Node 0: checkpoint 10 complete; stable timestamp = 2175
      Node 0: checkpoint 11 complete; stable timestamp = 2393
      Node 0: step-down checkpoint at 2470 (lsn 3519)
      Node 0: step-down emitted 739 events
      Node 0: no peer to hand over to; continuing alone
      Node 0: now follower
      Node 0: switch 2 complete
      [1787875174:186857][13536:0xffff72de70e0], checkpoint-pick-up: [WT_VERB_LAYERED][WARNING]: Picking up the same checkpoint again: metadata LSN = 3517
      Parent: directing switch 3
      Node 0: now leader
      Node 0: switch 3 complete
      Node 0: checkpoint 1 complete; stable timestamp = 4737
      Node 0: checkpoint 2 complete; stable timestamp = 4822
      Node 0: checkpoint 3 complete; stable timestamp = 5032
      Node 0: checkpoint 4 complete; stable timestamp = 5211
      Node 0: checkpoint 5 complete; stable timestamp = 5294
      Node 0: checkpoint 6 complete; stable timestamp = 5460
      Node 0: checkpoint 7 complete; stable timestamp = 5638
      Parent: directing switch 4
      Node 0: checkpoint 8 complete; stable timestamp = 5806
      Node 0: checkpoint 9 complete; stable timestamp = 5897
      Node 0: checkpoint 10 complete; stable timestamp = 5990
      Node 0: checkpoint 11 complete; stable timestamp = 6048
      Node 0: checkpoint 12 complete; stable timestamp = 6258
      Node 0: step-down checkpoint at 6314 (lsn 8093)
      Node 0: step-down emitted 737 events
      Node 0: no peer to hand over to; c
      

      logs

      schema-disagg-abort-follower-stepup-test-disagg-1 task_log

      Logs:

      [1787876566:890739][14567:0xffffae899020], file:schema_0_4_47_0.wt_stable, WT_CURSOR.search: [WT_VERB_DEFAULT][ERROR]: int __block_disagg_read_multiple(WT_SESSION_IMPL *, WT_BLOCK_DISAGG *, WT_PAGE_BLOCK_META *, uint64_t, uint64_t, uint64_t, uint64_t, uint32_t, uint32_t, WT_ITEM *, uint32_t *), 176: WiredTiger assertion failed: '*results_count > 0'. Expression returned false
      [1787876566:890792][14567:0xffffae899020], file:schema_0_4_47_0.wt_stable, WT_CURSOR.search: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 29: aborting WiredTiger library
      

      logs

      schema-disagg-abort-follower-stepup-test-disagg-1 task_log

      Logs:

      bash: line 61: 14567 Aborted                 (core dumped) ./test_schema_disagg_abort -b "$build_dir" -r f -s 30 -T 12 -u 64 -t 300
      Command 'shell.exec' in function 'schema disagg abort' (step 2 of 2) failed: shell script encountered problem: exit code 134.
      Finished command 'shell.exec' in function 'schema disagg abort' (step 2 of 2) in 25m2.261693005s.
      Running task commands failed: running command: command failed: shell script encountered problem: exit code 134
      Finished running task commands in 25m10.343987637s.
      Task completed - FAILURE.
      Running post-task commands.
      

      logs

      schema-disagg-abort-follower-stepup-test-disagg-1 task_log

      Logs:

      #0  0x0000ffffac4c4454 in ?? ()
      #0  0x0000ffffac4c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffffac47b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffffac462224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffffad101a5c [PAC] in __wt_abort (session=0xffff9cc0f960) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffffac8a2a18 in __block_disagg_read_multiple (session=0xffff9cc0f960, block_disagg=0x50b000015540, block_meta=0xffffaa95c630, page_id=104, flags=0, lsn=21465, base_lsn=0, size=65, checksum=2812075637, results_array=0xffffaa95c080, results_count=0xffffaa95c6a0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/block_disagg/block_disagg_read.c:176
      #5  0x0000ffffac8a1580 in __wti_block_disagg_read_multiple (bm=0x51400008c240, session=0xffff9cc0f960, block_meta=0xffffaa95c630, addr=0xffffaa8be2f0 "", addr_size=16, buffer_array=0xffffaa95c080, buffer_count=0xffffaa95c6a0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/block_disagg/block_disagg_read.c:361
      #6  0x0000ffffac871e58 in __wt_blkcache_read (session=0xffff9cc0f960, buf=0xffffaa46bf40, block_meta=0xffffaa46bfb0, addr=0xffffaa8be2e0 "", addr_size=16) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/block_cache/block_io.c:186
      #7  0x0000ffffac9ce8fc in __wti_btree_tree_open (session=0xffff9cc0f960, addr=0xffffaa8be2e0 "", addr_size=16) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/btree/bt_handle.c:950
      #8  0x0000ffffac9c6950 in __wt_btree_open (session=0xffff9cc0f960, op_cfg=0xffffaa3627a0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/btree/bt_handle.c:307
      #9  0x0000ffffacd06740 in __wt_conn_dhandle_open (session=0xffff9cc0f960, cfg=0xffffaa3627a0, flags=0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_dhandle.c:658
      #10 0x0000ffffad385630 in __wt_session_get_dhandle (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", checkpoint=0x0, cfg=0xffffaa3627a0, flags=0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/session/session_dhandle.c:1025
      #11 0x0000ffffad38554c in __wt_session_get_dhandle (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", checkpoint=0x0, cfg=0xffffaa3627a0, flags=0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/session/session_dhandle.c:1019
      #12 0x0000ffffad38266c in __wt_session_get_btree_ckpt (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", cfg=0xffffaa3627a0, flags=0, hs_dhandlep=0xffffaa7850c0, ckpt_snapshot=0xffffaa785020) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/session/session_dhandle.c:507
      #13 0x0000fffface066a0 in __wt_curfile_open (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", owner=0x51600004c880, cfg=0xffffaa3627a0, cursorp=0x51600004ca70) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_file.c:1333
      #14 0x0000ffffad30ace8 in __session_open_cursor_int (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", owner=0x51600004c880, other=0x0, cfg=0xffffaa3627a0, hash_value=1504160082667447094, cursorp=0x51600004ca70) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/session/session_api.c:697
      #15 0x0000ffffad309a88 in __wt_open_cursor (session=0xffff9cc0f960, uri=0x503000300490 "file:schema_0_4_47_0.wt_stable", owner=0x51600004c880, cfg=0xffffaa3627a0, cursorp=0x51600004ca70) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/session/session_api.c:806
      #16 0x0000ffffacebb9c8 in __clayered_open_stable_int (clayered=0x51600004c880, stable_uri=0x503000300490 "file:schema_0_4_47_0.wt_stable") at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:650
      #17 0x0000ffffaceba1cc in __clayered_open_stable_leader (clayered=0x51600004c880) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:887
      #18 0x0000ffffaceb9e2c in __clayered_open_stable (clayered=0x51600004c880, checkpoint_expected=false, role=WTI_CLAYERED_ROLE_LEADER) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:906
      #19 0x0000ffffaceb81d4 in __clayered_open_stable_first (clayered=0x51600004c880, role=WTI_CLAYERED_ROLE_LEADER, conn_lsn=0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:1221
      #20 0x0000ffffaceb606c in __clayered_update_stable (clayered=0x51600004c880, flags=36, role=WTI_CLAYERED_ROLE_LEADER) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:1252
      #21 0x0000ffffaceb3a54 in __clayered_enter (clayered=0x51600004c880, mode=WTI_CLAYERED_MODE_SEARCH, op=0xffffaa82b020) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:534
      #22 0x0000fffface90740 in __clayered_search (cursor=0x51600004c880) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/cursor/cur_layered.c:2499
      #23 0x0000aaaacd4d9838 in ?? ()
      #24 0x0000ffffc083fb60 in ?? ()
      

      logs

      schema-disagg-abort-follower-stepup-test-disagg-1 task_log

      Logs:

      Finished command 'shell.exec' in function 'dump stacktraces' (step 1 of 9) in block 'post' in 2.825986733s.
      Running command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/*stacktrace.txt]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post' in 176.481147ms.
      Running command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      File filter 'wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG' matched no files.
      Finished command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post' in 55.39515ms.
      Running command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      File filter 'wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload' matched no files.
      Finished command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post' in 47.470246ms.
      Running command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post'.
      if [ -d "WT_TEST" ]; then
        # Dump stderr/stdout contents generated by the C libraries onto console for Python tests
        find "WT_TEST" -name "std*.txt" ! -empty -exec bash -c "echo 'Contents from {}:'; cat '{}'" \;
      fi
      Finished command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post' in 1.386754ms.
      Running command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post'.
      mkdir stat_files
      python3 ../test/evergreen/collect_stat_files.py -d stat_files
      Finished command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post' in 62.299497ms.
      Running command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='4' target='/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger/cmake_build/stat_files.tar.gz']
      Finished command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' in 36.77311ms.
      Running command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post'.
      Putting local file 'wiredtiger/cmake_build/stat_files.tar.gz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post' in 452.167812ms.
      Running command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='8602' target='/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wiredtiger.tgz']
      Finished command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' in 1.427179968s.
      Running command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post'.
      Putting local file 'wiredtiger.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post' in 1.682124817s.
      Running command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post'.
      Beginning to build archive.
      No files were archived.
      Finished command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' in 348.373µs.
      Running command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post'.
      Putting local file 'wt-hang-analyzer.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-follower-stepup-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' in bucket 'build_external' (attempt 1 of 5).
      File '/data/mci/81813e47d5e7c6e92e7d5e1409ce8476/wt-hang-analyzer.tgz' not found and skip missing is true, exiting without error.
      Finished command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post' in 18.958599ms.
      Running command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post'.
      Finished command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post' in 399.678025ms.
      Finished running post-task commands in 7.187853857s.
      reading test log spec; falling back to default spec: open /data/mci/81813e47d5e7c6e92e7d5e1409ce8476/build/TestLogs/log_spec.yaml: no such file or directory
      

      logs

            Assignee:
            Jie Chen
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: