failed: schema-disagg-abort-multi-switch-crash-test-disagg-1 on amazon2023-disagg-asan-stress [wiredtiger-disagg @ 5cc20887]

XMLWordPrintableJSON

      schema-disagg-abort-multi-switch-crash-test-disagg-1 on amazon2023-disagg-asan-stress

      Host: i-03649fead05a28980
      Project: wiredtiger-disagg
      Commit: 5cc20887
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      Task logger initialized (agent version '2026-08-26' from Evergreen build revision 'a74e5e27fc4aa76b91a1c485a1613ca06d036c2f').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_switch_crash_test_disagg_1_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 3.615481ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 50.584µs.
      Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
      mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 already installed, skipping download.
      Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3.057988ms.
      Finished running pre-task commands in 7.106427ms.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 8.142144862s.
      Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
        [ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
        export TESTUTIL_TSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 0 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3619
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_switch_crash_test_disagg_1_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-128-107-71.ec2.internal
      INVOCATION_ID=aaf0676c594044d78c94e0d62aebe9ee
      JASPER_ID=8f01726d-bb16-4217-875a-12fa5b404139
      JASPER_MANAGER=cb1badc9-d84b-4dfc-af89-f0018889bba9
      JOURNAL_STREAM=8:27006
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      echo "Dump Environment"
      LOGNAME=ec2-user
      printenv | sort
      build_dir=$(cd ../../../ && pwd)
      LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/test/evergreen/asan_leaks.supp
      for i in $(seq 10); do
        echo "Iteration $i/10"
      MAIL=/var/spool/mail/ec2-user
        ./test_schema_disagg_abort -b "$build_dir" -r lf -s 30 -k l150 -T 12 -u 64 -t 300
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      done
      PWD=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3451
      S_COLORS=auto
      TEMP=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/tmp
      TESTUTIL_BYPASS_ASAN=1
      TMP=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/tmp
      TMPDIR=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      Iteration 1/10
      Running test command: ./test_schema_disagg_abort -b /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build -r lf -s 30 -k l150 -T 12 -u 64 -t 300
      Parent: roles lf; 12 schema threads; pool 64 slots; switch every 30s; kill leader@150 follower@0 lone@0; stop at 300s
      CONFIG: test_schema_disagg_abort -r lf -s 30 -k l150 -u 64 -T 12 -t 300 -PSD1285186,E5554258
      Running test command: /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A l -h WT_TEST.test_schema_disagg_abort -b /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build -T 12 -u 64 -s 30 -R 3 -W 6 -PSD1285186,E5554258
      Node 0: starting as leader
      Running test command: /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 1 -A f -h WT_TEST.test_schema_disagg_abort -b /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build -T 12 -u 64 -s 30 -R 5 -W 4 -PSD1285186,E5554258
      Node 1: starting as follower
      Node 0: checkpoint 1 complete; stable timestamp = 5
      Node 0: checkpoint 2 complete; stable timestamp = 20
      Node 0: checkpoint 3 complete; stable timestamp = 36
      Node 0: checkpoint 4 complete; stable timestamp = 55
      Node 0: checkpoint 5 complete; stable timestamp = 78
      Node 0: checkpoint 6 complete; stable timestamp = 154
      Node 0: checkpoint 7 complete; stable timestamp = 211
      Node 0: checkpoint 8 complete; stable timestamp = 258
      Node 0: checkpoint 9 complete; stable timestamp = 310
      Node 0: checkpoint 10 complete; stable timestamp = 379
      Parent: directing switch 1
      Node 0: checkpoint 11 complete; stable timestamp = 470
      Node 0: checkpoint 12 complete; stable timestamp = 518
      Node 0: checkpoint 13 complete; stable timestamp = 604
      Node 0: checkpoint 14 complete; stable timestamp = 772
      Node 0: step-down checkpoint at 886 (lsn 1655)
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      [1787875831:644629][592572:0xffff9bf0c020], disagg-step-up: [WT_VERB_DEFAULT][ERROR]: int __layered_create_missing_stable_tables_helper(WT_SESSION_IMPL *), 298: WiredTiger assertion failed: 'entry->schema_epoch > last_ckpt_epoch'. Expression returned false
      [1787875831:644690][592572:0xffff9bf0c020], disagg-step-up: [WT_VERB_DEFAULT][ERROR]: void __wt_abort(WT_SESSION_IMPL *), 29: aborting WiredTiger library
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      Node 0: step-down emitted 1439 events
      Node 0: now follower
      Node 0: peer died; continuing as a lone follower
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      test_schema_disagg_abort: FAILED: node1 terminated unexpectedly: signal 6: Invalid argument
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      Running task commands failed: running command: command failed: shell script encountered problem: exit code 134
      Finished running task commands in 1m16.234895665s.
      Task completed - FAILURE.
      Running post-task commands.
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      #0  0x0000ffff998c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff998c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff9987b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff99862224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9a501a5c [PAC] in __wt_abort (session=0x0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000aaaacb623560 in testutil_die (e=22, fmt=0xaaaacb63a740 <str> "%s terminated unexpectedly: %s %d") at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/utility/misc.c:75
      #5  0x0000aaaacb6149c4 in die_on_child_status (proc=0xaaaacb6d6c38 <run_children[children]+24>, status=SUBPROC_KILLED, code=6) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:140
      #6  0x0000aaaacb61410c in child_alive (proc=0xaaaacb6d6c38 <run_children[children]+24>) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:157
      #7  0x0000aaaacb613ee4 in children_poll (children=0xaaaacb6d6c20 <run_children[children]>) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:171
      #8  0x0000aaaacb613e0c in wait_for_sentinel (cfg=0xffff97e00020, children=0xaaaacb6d6c20 <run_children[children]>, sentinel=0xffff97701120 "switch_done.1", max_wait=240) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:186
      #9  0x0000aaaacb61479c in direct_switch (cfg=0xffff97e00020, children=0xaaaacb6d6c20 <run_children[children]>, cur_leaderp=0xffff979000a0, cur_followerp=0xffff979000b0, genp=0xffff979000c0) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:245
      #10 0x0000aaaacb611ef0 in run_children (cfg=0xffff97e00020, self_path=0xffff97e02190 "/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:342
      #11 0x0000aaaacb610f0c in parent_main (cfg=0xffff97e00020, self_path=0xffff97e02190 "/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/parent.c:407
      #12 0x0000aaaacb60a890 in main (argc=15, argv=0xffffd454f688) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/main.c:461
      #0  0x0000ffff99ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff99ac4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff99a7b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff99a62224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9a701a5c [PAC] in __wt_abort (session=0xffff98e9c840) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff9a36b720 in __layered_create_missing_stable_tables_helper (session=0xffff98e9c840) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_layered.c:298
      #5  0x0000ffff9a369470 in __layered_create_missing_stable_tables (session=0xffff98e9c840) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_layered.c:354
      #6  0x0000ffff9a361050 in __disagg_step_up (session=0xffff98e9a800) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_layered.c:1334
      #7  0x0000ffff9a35ca48 in __wti_disagg_conn_config (session=0xffff98e9a800, cfg=0xffff97ece020, reconfig=true) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_layered.c:1805
      #8  0x0000ffff9a389c3c in __wti_conn_reconfig (session=0xffff98e9a800, cfg=0xffff97ece020) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_reconfig.c:449
      #9  0x0000ffff9a2bb614 in __conn_reconfigure (wt_conn=0x525000000100, config=0xaaaacde99800 <str> "disaggregated=(role=leader)") at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/src/conn/conn_api.c:1488
      #10 0x0000aaaacde7052c in node_step_up (state=0xaaaacdecd340 <workload_state_create[state]>, final_ts=1450) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/node.c:340
      #11 0x0000aaaacde6ea5c in node_run (cfg=0xffff98000020, state=0xaaaacdecd340 <workload_state_create[state]>, role=0xaaaacdebfb80 <node_role.node_role_follower>) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/node.c:390
      #12 0x0000aaaacde6e2e0 in node_main (cfg=0xffff98000020) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/node.c:438
      #13 0x0000aaaacde6a7dc in main (argc=22, argv=0xffffde3d1c38) at /data/mci/0994907b28436b0232717ec4c5d7d8a7/wiredtiger/test/csuite/schema_disagg_abort/main.c:450
      

      logs

      schema-disagg-abort-multi-switch-crash-test-disagg-1 task_log

      Logs:

      Finished command 'shell.exec' in function 'dump stacktraces' (step 1 of 9) in block 'post' in 3.560002252s.
      Running command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/*stacktrace.txt]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post' in 205.397456ms.
      Running command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      File filter 'wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG' matched no files.
      Finished command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post' in 55.040075ms.
      Running command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload]' into path 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/' in bucket 'build_external' (attempt 1 of 5).
      File filter 'wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload' matched no files.
      Finished command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post' in 45.57104ms.
      Running command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post'.
      if [ -d "WT_TEST" ]; then
        # Dump stderr/stdout contents generated by the C libraries onto console for Python tests
        find "WT_TEST" -name "std*.txt" ! -empty -exec bash -c "echo 'Contents from {}:'; cat '{}'" \;
      fi
      Finished command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post' in 1.477484ms.
      Running command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post'.
      mkdir stat_files
      python3 ../test/evergreen/collect_stat_files.py -d stat_files
      Finished command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post' in 46.405231ms.
      Running command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='4' target='/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger/cmake_build/stat_files.tar.gz']
      Finished command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' in 16.386933ms.
      Running command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post'.
      Putting local file 'wiredtiger/cmake_build/stat_files.tar.gz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12/stat_files.tar.gz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post' in 338.397526ms.
      Running command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='8565' target='/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wiredtiger.tgz']
      Finished command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' in 1.450641575s.
      Running command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post'.
      Putting local file 'wiredtiger.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/artifacts/schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12-0.tgz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post' in 1.480679021s.
      Running command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post'.
      Beginning to build archive.
      No files were archived.
      Finished command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' in 373.292µs.
      Running command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post'.
      Putting local file 'wt-hang-analyzer.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/5cc20887634b37b29727ebb61852236650d8ef12/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-multi-switch-crash-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_5cc20887634b37b29727ebb61852236650d8ef12_26_08_27_22_23_12.tgz' in bucket 'build_external' (attempt 1 of 5).
      File '/data/mci/1667b0afe3fc268f50a7abb2d9ed185a/wt-hang-analyzer.tgz' not found and skip missing is true, exiting without error.
      Finished command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post' in 13.119946ms.
      Running command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post'.
      Finished command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post' in 445.371619ms.
      Finished running post-task commands in 7.660457397s.
      reading test log spec; falling back to default spec: open /data/mci/1667b0afe3fc268f50a7abb2d9ed185a/build/TestLogs/log_spec.yaml: no such file or directory
      

      logs

            Assignee:
            Jie Chen
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: