test/format (mode=switch+fast_truncate) WiredTiger assertion failed: __rec_append_orig_value

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Disagg CI-blocker

      format-stress-test-disagg-switch-fast-truncate-1 on amazon2023-disagg-stress

      Host: i-0ffdbf9b65d4fd251
      Project: wiredtiger-disagg
      Commit: 31faf070
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-fast-truncate-1 task_log

      Logs:

      Task logger initialized (agent version '2026-06-02' from Evergreen build revision '8a1c4e5a57382ad1bf33192f251bcdacaac2c80d').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_fast_truncate_1_31faf07090cca0bc915e615b662dc41ce08e36ff_26_06_04_02_56_06', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre' in 2.40745ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre' in 38.474µs.
      Finished running pre-task commands in 2.626859ms.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-stress/31faf07090cca0bc915e615b662dc41ce08e36ff/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_stress_31faf07090cca0bc915e615b662dc41ce08e36ff_26_06_04_02_56_06.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 5.812979621s.
      Running command 'shell.exec' in function 'format test disagg' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
        [ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_TSAN=1
      elif [[ "" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 1 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3522
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_fast_truncate_1_31faf07090cca0bc915e615b662dc41ce08e36ff_26_06_04_02_56_06
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger/cmake_build/test/format/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-122-54-139.ec2.internal
      INVOCATION_ID=4c6cfb6d568a4afdb1e90a7f1f4ca779
      JASPER_ID=c66cc53a-38d6-4e54-abfd-0e68db507b1d
      JASPER_MANAGER=58794a50-5f3b-4eb9-8b98-350e49d0d496
      JOURNAL_STREAM=8:17789
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger/cmake_build
      LD_PRELOAD=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger/TCMALLOC_LIB/libtcmalloc.so
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      LOGNAME=ec2-user
      MAIL=/var/spool/mail/ec2-user
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      PWD=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger/cmake_build/test/format
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3400
      S_COLORS=auto
      TEMP=/data/mci/e08d51252bd1629c29764bbf21c79a23/tmp
      TMP=/data/mci/e08d51252bd1629c29764bbf21c79a23/tmp
      TMPDIR=/data/mci/e08d51252bd1629c29764bbf21c79a23/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/e08d51252bd1629c29764bbf21c79a23/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      echo "Dump Environment"
      printenv | sort
      # Fail, show the configuration file.
      fail() {
        echo "======= FAILURE =========="
        [ -f RUNDIR/CONFIG ] && cat RUNDIR/CONFIG
        exit 1
      }
      for i in $(seq 5); do
        echo Iteration $i/5
        rm -rf RUNDIR
        ./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch debug.disagg_slow_truncate_follower=off runs.timer=10:15 ops.prepare=0 ops.verify=1 runs.mirror=1  || fail
        if [ true = true ]; then
          ./t -R $format_args  || fail
        fi
      done
      

      logs

      format-stress-test-disagg-switch-fast-truncate-1 task_log

      Logs:

      [1780547522:990795][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __rec_append_orig_value, 219: WiredTiger assertion failed: '(unpack->tw.stop_ts == oldest_upd->u.commit.start_ts || unpack->tw.stop_ts == 0) && (unpack->tw.stop_txn == oldest_upd->txnid || unpack->tw.stop_txn == 0 || (oldest_upd->txnid == 0 && ((((conn)->flags) & (0x0800u)) != 0) && ((((oldest_upd)->flags) & (0x080u)) != 0)))'. Expression returned false
      [1780547522:990818][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      [1780547522:990830][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990836][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990838][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990841][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990842][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990844][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990846][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990848][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990850][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990851][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990853][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990855][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990857][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990859][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990861][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990862][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990866][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_next, 321: Error at src/cursor/cur_hs.c:321: "__curhs_next_visible(session, hs_cursor)" failed: WT_NOTFOUND: item not found
      [1780547522:990868][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990870][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990871][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990873][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990875][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990877][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990879][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990880][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990882][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990884][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990886][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990888][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990889][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990891][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990894][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990895][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990897][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990899][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990901][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_search_near_helper, 745: Error at src/cursor/cur_hs.c:745: "cursor->search_near(cursor, &cmp)" failed: WT_NOTFOUND: item not found
      [1780547522:990903][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curhs_searc
      

      logs

      format-stress-test-disagg-switch-fast-truncate-1 task_log

      Logs:

      Iteration 1/5
      t: process 23969 running
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1046465,metadata_checksum=aa5b52bd,database_size=799906346,version=1,compatible_version=1],timestamp=0x78483f) ---
      0x31ee0de5f000:[1780547522:990823][23969:0xffff959ddbc0], t, file:T00003.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][WARNING]: 172591 errors occurred, only the last 100 are shown
      bash: line 67: 23969 Aborted                 (core dumped) ./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch debug.disagg_slow_truncate_follower=off runs.timer=10:15 ops.prepare=0 ops.verify=1 runs.mirror=1
      ======= FAILURE ==========
      ############################################
      #  RUN PARAMETERS: V3
      ############################################
      assert.read_timestamp=0
      background_compact=0
      background_compact.free_space_target=4
      backup=0
      backup.incremental=off
      backup.incr_granularity=12795
      backup.live_restore=0
      backup.live_restore_read_size=8
      backup.live_restore_threads=7
      block_cache=0
      block_cache.cache_on_checkpoint=0
      block_cache.cache_on_writes=1
      block_cache.size=83
      cache=3072
      cache.evict_max=0
      cache.eviction_dirty_target=0
      cache.eviction_dirty_trigger=0
      cache.eviction_updates_target=0
      cache.eviction_updates_trigger=0
      cache.minimum=0
      cache.maximum=0
      checkpoint=on
      checkpoint.log_size=166
      checkpoint.wait=12
      checkpoint_threads=2
      compact.free_space_target=43
      debug.background_compact=1
      debug.checkpoint_retention=9
      debug.cursor_reposition=0
      debug.disagg_slow_truncate_follower=0
      debug.eviction=0
      debug.log_retention=7
      debug.realloc_exact=0
      debug.realloc_malloc=0
      debug.slow_checkpoint=0
      debug.slow_truncate=0
      debug.table_logging=0
      debug.update_restore_evict=0
      disagg.internal_page_delta=0
      disagg.leaf_page_delta=1
      disagg.multi=0
      disagg.multi_validation=0
      disagg.enabled=1
      disagg.layered=1
      disagg.mode=switch
      disagg.page_log=palite
      disagg.key_provider=0
      disagg.page_log.verbose=0
      disagg.drain_threads=7
      disagg.preserve=0
      disk.data_extend=0
      disk.encryption=none
      disk.mmap=1
      disk.mmap_all=0
      eviction.evict_use_softptr=0
      file_manager.close_handle_minimum=25
      file_manager.close_idle_time=42
      file_manager.close_scan_interval=8
      format.abort=0
      format.independent_thread_rng=1
      format.major_timeout=0
      import=0
      logging=0
      logging.compression=none
      logging.file_max=94406
      logging.prealloc=1
      logging.remove=1
      obsolete_cleanup.method=off
      obsolete_cleanup.wait=1504
      ops.alter=0
      ops.compaction=0
      ops.hs_cursor=0
      ops.bound_cursor=0
      ops.prepare=0
      ops.reserve=13
      ops.random_cursor=0
      ops.salvage=0
      ops.throttle=0
      ops.throttle.sleep_us=957500
      ops.verify=1
      prefetch=0
      prefetch.default=0
      precise_checkpoint=1
      preserve_prepared=0
      quiet=1
      random.data_seed=14853690
      random.extra_seed=8408667
      rollback_to_stable_threads=3
      runs.in_memory=0
      runs.mirror=1
      runs.ops=0
      runs.predictable_replay=0
      runs.source=layered
      runs.tables=3
      runs.threads=19
      runs.timer=11
      runs.type=row-store
      runs.verify_failure_dump=0
      statistics.mode=all
      statistics_log.sources=off
      stress.aggressive_stash_free=0
      stress.aggressive_sweep=0
      stress.checkpoint=0
      stress.checkpoint_evict_page=0
      stress.checkpoint_prepare=0
      stress.compact_slow=0
      stress.evict_reposition=0
      stress.failpoint_eviction_split=0
      stress.failpoint_hs_delete_key_from_ts=1
      stress.failpoint_rec_before_wrapup=0
      stress.hs_checkpoint_delay=0
      stress.hs_search=0
      stress.hs_sweep=0
      stress.prefetch_delay=0
      stress.prepare_resolution_1=0
      stress.sleep_before_read_overflow_onpage=0
      stress.split_1=0
      stress.split_2=0
      stress.split_3=0
      stress.split_4=0
      stress.split_5=0
      stress.split_6=0
      stress.split_7=0
      stress.split_8=0
      tiered_storage.flush_frequency=0
      tiered_storage.storage_source=off
      transaction.implicit=0
      transaction.operation_timeout_ms=2000
      transaction.timestamps=1
      wiredtiger.config=off
      wiredtiger.rwlock=1
      wiredtiger.leak_memory=0
      ############################################
      #  TABLE PARAMETERS: table 1
      ############################################
      table1.btree.compression=none
      table1.btree.dictionary=0
      table1.btree.internal_key_truncation=1
      table1.btree.internal_page_max=12
      table1.btree.key_max=102
      table1.btree.key_min=32
      table1.btree.leaf_page_max=15
      table1.btree.memory_page_max=4
      table1.btree.prefix_len=0
      table1.btree.prefix_compression=1
      table1.btree.prefix_compression_min=7
      table1.btree.reverse=0
      table1.btree.split_pct=61
      table1.btree.value_max=4049
      table1.btree.value_min=17
      table1.disk.checksum=uncompressed
      table1.disk.firstfit=0
      table1.ops.pareto=0
      table1.ops.pareto.skew=41
      table1.ops.pct.delete=14
      table1.ops.pct.insert=26
      table1.ops.pct.modify=2
      table1.ops.pct.read=18
      table1.ops.pct.write=40
      table1.ops.truncate=1
      table1.runs.rows=135285
      ############################################
      #  TABLE PARAMETERS: table 2
      ############################################
      table2.btree.compression=none
      table2.btree.dictionary=0
      table2.btree.internal_key_truncation=1
      table2.btree.internal_page_max=12
      table2.btree.key_max=115
      table2.btree.key_min=23
      table2.btree.leaf_page_max=13
      table2.btree.memory_page_max=9
      table2.btree.prefix_len=0
      table2.btree.prefix_compression=0
      table2.btree.prefix_compression_min=2
      table2.btree.reverse=0
      table2.btree.split_pct=50
      table2.btree.value_max=1768
      table2.btree.value_min=13
      table2.disk.checksum=uncompressed
      table2.disk.firstfit=1
      table2.ops.pareto=1
      table2.ops.pareto.skew=70
      table2.ops.pct.delete=9
      table2.ops.pct.insert=62
      table2.ops.pct.modify=1
      table2.ops.pct.read=26
      table2.ops.pct.write=2
      table2.ops.truncate=1
      table2.runs.rows=135285
      ############################################
      #  TABLE PARAMETERS: table 3
      ############################################
      table3.btree.compression=none
      table3.btree.dictionary=0
      table3.btree.internal_key_truncation=1
      table3.btree.internal_page_max=13
      table3.btree.key_max=70
      table3.btree.key_min=28
      table3.btree.leaf_page_max=16
      table3.btree.memory_page_max=4
      table3.btree.prefix_len=0
      table3.btree.prefix_compression=1
      table3.btree.prefix_compression_min=5
      table3.btree.reverse=0
      table3.btree.split_pct=99
      table3.btree.value_max=2738
      table3.btree.value_min=20
      table3.disk.checksum=uncompressed
      table3.disk.firstfit=0
      table3.ops.pareto=1
      table3.ops.pareto.skew=70
      table3.ops.pct.delete=29
      table3.ops.pct.insert=0
      table3.ops.pct.modify=57
      table3.ops.pct.read=12
      table3.ops.pct.write=2
      table3.ops.truncate=1
      table3.runs.rows=135285
      Command 'shell.exec' in function 'format test disagg' (step 2 of 2) failed: shell script encountered problem: exit code 1.
      Finished command 'shell.exec' in function 'format test disagg' (step 2 of 2) in 4m9.415349672s.
      Running task commands failed: running command: command failed: shell script encountered problem: exit code 1
      Finished running task commands in 4m15.228643802s.
      Task completed - FAILURE.
      Running post-task commands.
      

      logs

      format-stress-test-disagg-switch-fast-truncate-1 task_log

      Logs:

      #0  0x0000ffff972c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff972c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff9727b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff97262224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9764f05c [PAC] in __wt_abort (session=session@entry=0x31ee0de5f000) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffff97690128 in __rec_append_orig_value (session=session@entry=0x31ee0de5f000, page=page@entry=0x31eedb681b00, upd=upd@entry=0x31ee05153ec0, unpack=unpack@entry=0xffff959db158, write_prepared=write_prepared@entry=false) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_visibility.c:219
      #5  0x0000ffff97690b08 in __rec_fill_tw_from_upd_select (session=session@entry=0x31ee0de5f000, page=page@entry=0x31eedb681b00, vpack=vpack@entry=0xffff959db158, upd_select=upd_select@entry=0xffff959db0d8, write_prepare=<optimized out>, r=r@entry=0x31eec4a04c80) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_visibility.c:1427
      #6  0x0000ffff97691218 in __wti_rec_upd_select (session=session@entry=0x31ee0de5f000, r=r@entry=0x31eec4a04c80, ins=ins@entry=0x0, rip=rip@entry=0x31eedb681f58, vpack=vpack@entry=0xffff959db158, upd_select=upd_select@entry=0xffff959db0d8) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_visibility.c:1563
      #7  0x0000ffff97686904 in __wti_rec_row_leaf (session=session@entry=0x31ee0de5f000, r=r@entry=0x31eec4a04c80, pageref=pageref@entry=0x31ee0949a1e0, salvage=salvage@entry=0x0) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_row.c:1144
      #8  0x0000ffff9769e8f8 in __reconcile (session=session@entry=0x31ee0de5f000, ref=ref@entry=0x31ee0949a1e0, salvage=salvage@entry=0x0, flags=flags@entry=672, page_lockedp=page_lockedp@entry=0xffff959dbb2f) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_write.c:316
      #9  0x0000ffff9769da2c in __wt_reconcile (session=session@entry=0x31ee0de5f000, ref=ref@entry=0x31ee0949a1e0, salvage=salvage@entry=0x0, flags=flags@entry=672) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/reconcile/rec_write.c:127
      #10 0x0000ffff97600560 in __evict_reconcile (session=session@entry=0x31ee0de5f000, ref=ref@entry=0x31ee0949a1e0, evict_flags=evict_flags@entry=0) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_page.c:1277
      #11 0x0000ffff97602ca4 in __wt_evict (session=session@entry=0x31ee0de5f000, ref=0x31ee0949a1e0, previous_state=3 '\003', flags=flags@entry=0) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_page.c:443
      #12 0x0000ffff975f7a84 in __wti_evict_page (session=session@entry=0x31ee0de5f000, is_server=is_server@entry=true) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_dispatch.c:254
      #13 0x0000ffff97603984 in __wti_evict_lru_pages (session=session@entry=0x31ee0de5f000, is_server=is_server@entry=true) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_queue.c:140
      #14 0x0000ffff9760738c in __evict_pass (session=session@entry=0x31ee0de5f000) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_thread.c:696
      #15 0x0000ffff976074cc in __evict_server (session=session@entry=0x31ee0de5f000, did_work=did_work@entry=0xffff959dce9f) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_thread.c:198
      #16 0x0000ffff97607964 in __evict_thread_run (session=0x31ee0de5f000, thread=0x31ee2a1fd2c0) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/evict/evict_thread.c:93
      #17 0x0000ffff97715944 in __thread_run (arg=0x31ee2a1fd2c0) at /data/mci/dbf4bff9ed0a1fe0934f97fe94b672c7/wiredtiger/src/support/thread_group.c:32
      #18 0x0000ffff972c2834 in start_thread () from /lib64/libc.so.6
      #19 0x0000ffff97266e5c [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

            Assignee:
            Unassigned
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: