test/format (disagg.mode=switch) mirror mismatch

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: Disagg CI-blocker

      format-stress-test-disagg-switch-data-validation-1 on amazon2023-disagg-stress

      Host: i-09fe0f93812264758
      Project: wiredtiger-disagg
      Commit: ae12b501
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      Task logger initialized (agent version '2026-05-01a' from Evergreen build revision '506cfa5dd480356b3052925f267bd0aabe7d8ec5').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_1_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre' in 2.436324ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre' in 33.634µs.
      Finished running pre-task commands in 2.658193ms.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 10.066249522s.
      Running command 'shell.exec' in function 'format test disagg' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname $(/opt/mongodbtoolchain/v5/bin/clang -print-file-name=libclang_rt.asan.so))"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_TSAN=1
      elif [[ "" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 1 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3585
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_1_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/cmake_build/test/format/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-122-63-141.ec2.internal
      INVOCATION_ID=3a3f78f00ee54b7abc9e0bedb7e09a5f
      JASPER_ID=71ed47d7-58bb-4bfe-9282-bd3e487facf0
      JASPER_MANAGER=46de9185-063b-4568-9c54-35d5c08bb404
      JOURNAL_STREAM=8:17960
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/cmake_build
      LD_PRELOAD=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/TCMALLOC_LIB/libtcmalloc.so
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      LOGNAME=ec2-user
      MAIL=/var/spool/mail/ec2-user
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      PWD=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/cmake_build/test/format
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3360
      S_COLORS=auto
      TEMP=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/tmp
      TMP=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/tmp
      TMPDIR=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      echo "Dump Environment"
      printenv | sort
      # Fail, show the configuration file.
      fail() {
        echo "======= FAILURE =========="
        [ -f RUNDIR/CONFIG ] && cat RUNDIR/CONFIG
        exit 1
      }
      for i in $(seq 5); do
        echo Iteration $i/5
        rm -rf RUNDIR
        ./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch ops.verify=1 runs.mirror=1 table1.runs.source=table table1.disagg.enabled=0 runs.timer=5:10 ops.prepare=1  || fail
        if [ true = true ]; then
          ./t -R $format_args  || fail
        fi
      done
      t: WARNING: table.1 skipped verify because of EBUSY
      t: WARNING: table.2 skipped verify because of EBUSY
      t: WARNING: table.3 skipped verify because of EBUSY
      Iteration 1/5
      t: process 3856 running
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=368494,metadata_checksum=a621af82,database_size=532024647,version=1,compatible_version=1],timestamp=0x59fa5b) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=714367,metadata_checksum=b617fdd4,database_size=523180955,version=1,compatible_version=1],timestamp=0x7b3920) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1003618,metadata_checksum=fbcb54ce,database_size=524973159,version=1,compatible_version=1],timestamp=0x991f5d) ---
      t: successful run completed (485 seconds)
      t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls
      t: WARNING: turning off ops.truncate to work with ops.prepare and precise checkpoint
      mirror: 153087/153087 mismatch: table:T00001: {0000153087.00/opqrstuvwxy}/{0000153087/LMNO}, table:T00002: {0000153087.00/opqrstuvwxyzabcdefghijklmnopqrstuv}/{0000153087/LMNOPQRSTUVWXYZABCD}
      last successful match was 153086
      

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      mirror error: base cursor (table 1): dumping to RUNDIR/FAIL.pagedump.1
      

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      mirror error: table cursor (table 2): dumping to RUNDIR/FAIL.pagedump.2
      

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      mirror: 166457/166457 mismatch: table:T00001: {0000166457.00/opqrstuvwxy}/{0000166457/LMNOPQRSTUVWXYZABC}, table:T00002: {0000166457.00/opqrstuvwxyzabcdefghijklmnopqrstu}/{0000166457/LMNOPQRSTUVWXY}
      last successful match was 166456
      mirror: 432129/432129 mismatch: table:T00001: {0000432129.00/o}/{0000432129/L}, table:T00002: {0000432129.00/opqrstuvwxyzabcdefghijklmno}/{0000432129/LMNOPQR}
      last successful match was 432128
      mirror: 553885/553885 mismatch: table:T00001: {0000553885.00/opqrstuvwxyza}/{0000553885/LMNOPQRSTUVWXY}, table:T00002: {0000553885.00/opqrstuvwxyzabcdefghijkl}/{0000553885/LMN}
      last successful match was 553884
      t: FAILED: table_verify_mirror/364: failures == 0
      t: run FAILED
      t: process aborting
      WiredTiger Error: __wt_abort, 29: aborting WiredTiger library
      bash: line 65:  4377 Aborted                 (core dumped) ./t -R $format_args
       t: process 4377 running
      ======= FAILURE ==========
      ############################################
      #  RUN PARAMETERS: V3
      ############################################
      assert.read_timestamp=0
      background_compact=0
      background_compact.free_space_target=28
      backup=0
      backup.incremental=off
      backup.incr_granularity=7454
      backup.live_restore=1
      backup.live_restore_read_size=2
      backup.live_restore_threads=12
      block_cache=0
      block_cache.cache_on_checkpoint=1
      block_cache.cache_on_writes=0
      block_cache.size=25
      cache=3072
      cache.evict_max=1
      cache.eviction_dirty_target=0
      cache.eviction_dirty_trigger=0
      cache.eviction_updates_target=0
      cache.eviction_updates_trigger=0
      cache.minimum=0
      cache.maximum=0
      checkpoint=on
      checkpoint.log_size=88
      checkpoint.wait=55
      checkpoint_threads=1
      compact.free_space_target=46
      debug.background_compact=0
      debug.checkpoint_retention=2
      debug.cursor_reposition=0
      debug.eviction=0
      debug.log_retention=4
      debug.realloc_exact=0
      debug.realloc_malloc=0
      debug.slow_checkpoint=0
      debug.table_logging=0
      debug.update_restore_evict=0
      disagg.internal_page_delta=1
      disagg.leaf_page_delta=0
      disagg.multi=0
      disagg.multi_validation=0
      disagg.enabled=1
      disagg.layered=1
      disagg.mode=switch
      disagg.page_log=palite
      disagg.key_provider=1
      disagg.page_log.verbose=0
      disagg.drain_threads=15
      disagg.preserve=0
      disk.data_extend=0
      disk.encryption=none
      disk.mmap=1
      disk.mmap_all=0
      eviction.evict_use_softptr=0
      file_manager.close_handle_minimum=13
      file_manager.close_idle_time=39
      file_manager.close_scan_interval=3
      format.abort=0
      format.independent_thread_rng=1
      format.major_timeout=0
      import=0
      logging=0
      logging.compression=none
      logging.file_max=17511
      logging.prealloc=0
      logging.remove=0
      obsolete_cleanup.method=off
      obsolete_cleanup.wait=960
      ops.alter=0
      ops.compaction=0
      ops.hs_cursor=1
      ops.bound_cursor=1
      ops.prepare=1
      ops.reserve=17
      ops.random_cursor=0
      ops.salvage=0
      ops.throttle=0
      ops.throttle.sleep_us=901559
      ops.verify=1
      prefetch=0
      prefetch.default=0
      precise_checkpoint=1
      preserve_prepared=1
      quiet=1
      random.data_seed=12738537
      random.extra_seed=435572
      rollback_to_stable_threads=5
      runs.in_memory=0
      runs.mirror=1
      runs.ops=0
      runs.predictable_replay=0
      runs.source=layered
      runs.tables=3
      runs.threads=30
      runs.timer=6
      runs.type=row-store
      runs.verify_failure_dump=0
      statistics.mode=all
      statistics_log.sources=off
      stress.aggressive_stash_free=0
      stress.aggressive_sweep=0
      stress.checkpoint=0
      stress.checkpoint_evict_page=0
      stress.checkpoint_prepare=0
      stress.compact_slow=0
      stress.evict_reposition=0
      stress.failpoint_eviction_split=1
      stress.failpoint_hs_delete_key_from_ts=0
      stress.failpoint_rec_before_wrapup=0
      stress.hs_checkpoint_delay=0
      stress.hs_search=0
      stress.hs_sweep=0
      stress.prefetch_delay=0
      stress.prepare_resolution_1=0
      stress.sleep_before_read_overflow_onpage=0
      stress.split_1=0
      stress.split_2=0
      stress.split_3=0
      stress.split_4=0
      stress.split_5=0
      stress.split_6=0
      stress.split_7=0
      stress.split_8=0
      tiered_storage.flush_frequency=0
      tiered_storage.storage_source=off
      transaction.implicit=0
      transaction.operation_timeout_ms=2000
      transaction.timestamps=1
      wiredtiger.config=off
      wiredtiger.rwlock=1
      wiredtiger.leak_memory=0
      ############################################
      #  TABLE PARAMETERS: table 1
      ############################################
      table1.btree.compression=none
      table1.btree.dictionary=0
      table1.btree.internal_key_truncation=1
      table1.btree.internal_page_max=14
      table1.btree.key_max=41
      table1.btree.key_min=13
      table1.btree.leaf_page_max=13
      table1.btree.memory_page_max=9
      table1.btree.prefix_len=0
      table1.btree.prefix_compression=1
      table1.btree.prefix_compression_min=0
      table1.btree.reverse=0
      table1.btree.split_pct=99
      table1.btree.value_max=633
      table1.btree.value_min=8
      table1.disagg.enabled=0
      table1.disk.checksum=on
      table1.disk.firstfit=0
      table1.ops.pareto=0
      table1.ops.pareto.skew=32
      table1.ops.pct.delete=0
      table1.ops.pct.insert=79
      table1.ops.pct.modify=3
      table1.ops.pct.read=5
      table1.ops.pct.write=13
      table1.ops.truncate=1
      table1.runs.rows=584706
      table1.runs.source=table
      ############################################
      #  TABLE PARAMETERS: table 2
      ############################################
      table2.btree.compression=none
      table2.btree.dictionary=0
      table2.btree.internal_key_truncation=1
      table2.btree.internal_page_max=17
      table2.btree.key_max=50
      table2.btree.key_min=31
      table2.btree.leaf_page_max=14
      table2.btree.memory_page_max=5
      table2.btree.prefix_len=0
      table2.btree.prefix_compression=1
      table2.btree.prefix_compression_min=4
      table2.btree.reverse=0
      table2.btree.split_pct=77
      table2.btree.value_max=1565
      table2.btree.value_min=6
      table2.disk.checksum=unencrypted
      table2.disk.firstfit=0
      table2.ops.pareto=0
      table2.ops.pareto.skew=48
      table2.ops.pct.delete=13
      table2.ops.pct.insert=0
      table2.ops.pct.modify=1
      table2.ops.pct.read=0
      table2.ops.pct.write=86
      table2.ops.truncate=1
      table2.runs.rows=584706
      ############################################
      #  TABLE PARAMETERS: table 3
      ############################################
      table3.btree.compression=none
      table3.btree.dictionary=1
      table3.btree.internal_key_truncation=1
      table3.btree.internal_page_max=13
      table3.btree.key_max=128
      table3.btree.key_min=15
      table3.btree.leaf_page_max=16
      table3.btree.memory_page_max=5
      table3.btree.prefix_len=0
      table3.btree.prefix_compression=1
      table3.btree.prefix_compression_min=1
      table3.btree.reverse=0
      table3.btree.split_pct=74
      table3.btree.value_max=1208
      table3.btree.value_min=11
      table3.disk.checksum=off
      table3.disk.firstfit=0
      table3.ops.pareto=0
      table3.ops.pareto.skew=78
      table3.ops.pct.delete=1
      table3.ops.pct.insert=2
      table3.ops.pct.modify=0
      table3.ops.pct.read=1
      table3.ops.pct.write=96
      table3.ops.truncate=1
      table3.runs.rows=584706
      Command 'shell.exec' in function 'format test disagg' (step 2 of 2) failed: shell script encountered problem: exit code 1.
      Finished command 'shell.exec' in function 'format test disagg' (step 2 of 2) in 8m13.64681636s.
      Running task commands failed: running command: command failed: shell script encountered problem: exit code 1
      Finished running task commands in 8m23.713443316s.
      Task completed - FAILURE.
      Running post-task commands.
      

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      #0  0x0000ffff926c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff926c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff9267b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff92662224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff92a463c4 [PAC] in __wt_abort (session=session@entry=0x0) at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000000000425990 in testutil_die (e=e@entry=0, fmt=fmt@entry=0x427db0 "%s/%d: %s") at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/test/utility/misc.c:75
      #5  0x000000000041d9d0 in table_verify_mirror (conn=conn@entry=0x73a37fc1c000, base=0x73a37fc16000, table=0x73a37fc18000, checkpoint=checkpoint@entry=0x0, tinfo=tinfo@entry=0x0) at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/test/format/verify.c:364
      #6  0x000000000041dc64 in wts_verify_mirrors (conn=0x73a37fc1c000, checkpoint=0x0, tinfo=tinfo@entry=0x0) at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/test/format/verify.c:452
      #7  0x000000000040d6b4 in wts_prepare_discover (conn=<optimized out>) at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/test/format/format_prepare_discover.c:121
      #8  0x000000000041b948 in main (argc=<optimized out>, argv=<optimized out>) at /data/mci/95713e0122669ecef228575fb2c64bbe/wiredtiger/test/format/t.c:364
      

      logs

      format-stress-test-disagg-switch-data-validation-1 task_log

      Logs:

      Finished command 'shell.exec' in function 'dump stacktraces' (step 1 of 9) in block 'post' in 3.752098151s.
      Running command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/*stacktrace.txt]' into path 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post' in 147.943891ms.
      Running command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG]' into path 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post' in 170.868093ms.
      Running command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post'.
      Putting files matching filter '[wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload]' into path 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in S3 bucket 'build_external'.
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/' in bucket 'build_external' (attempt 1 of 5).
      File filter 'wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload' matched no files.
      Finished command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post' in 43.521199ms.
      Running command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post'.
      if [ -d "WT_TEST" ]; then
        # Dump stderr/stdout contents generated by the C libraries onto console for Python tests
        find "WT_TEST" -name "std*.txt" ! -empty -exec bash -c "echo 'Contents from {}:'; cat '{}'" \;
      fi
      Finished command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post' in 1.349397ms.
      Running command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post'.
      mkdir stat_files
      python3 ../test/evergreen/collect_stat_files.py -d stat_files
      Finished command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post' in 35.140921ms.
      Running command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='2' target='/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger/cmake_build/stat_files.tar.gz']
      Finished command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' in 12.094514ms.
      Running command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post'.
      Putting local file 'wiredtiger/cmake_build/stat_files.tar.gz' into path 'build_external/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/stat_files.tar.gz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/stat_files.tar.gz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00/stat_files.tar.gz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post' in 232.670692ms.
      Running command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post'.
      Beginning to build archive.
      [message='successfully created archive' num_files='6974' target='/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wiredtiger.tgz']
      Finished command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' in 8.22142595s.
      Running command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post'.
      Putting local file 'wiredtiger.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00-0.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00-0.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/artifacts/format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00-0.tgz' in bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post' in 6.519562474s.
      Running command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post'.
      Beginning to build archive.
      No files were archived.
      Finished command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' in 339.201µs.
      Running command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post'.
      Putting local file 'wt-hang-analyzer.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/wt_hang_analyzer/wt-hang-analyzer_format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/wt_hang_analyzer/wt-hang-analyzer_format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00.tgz).
      Performing S3 put to file 'wiredtiger/amazon2023-disagg-stress/ae12b5013934cf7537fe09571403cca252a97cd5/wt_hang_analyzer/wt-hang-analyzer_format-stress-test-disagg-switch-data-validation-1_wiredtiger_disagg_amazon2023_disagg_stress_ae12b5013934cf7537fe09571403cca252a97cd5_26_05_07_22_48_00.tgz' in bucket 'build_external' (attempt 1 of 5).
      File '/data/mci/a50e98d34b2996bc6a216bdf25798fa2/wt-hang-analyzer.tgz' not found and skip missing is true, exiting without error.
      Finished command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post' in 20.497278ms.
      Running command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post'.
      Finished command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post' in 2.265439194s.
      Finished running post-task commands in 21.42439497s.
      reading test log spec; falling back to default spec: open /data/mci/a50e98d34b2996bc6a216bdf25798fa2/build/TestLogs/log_spec.yaml: no such file or directory
      

      logs

            Assignee:
            Unassigned
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: