Schema disagg abort __wt_ckpt_last_name failed: WT_NOTFOUND: item not found

XMLWordPrintableJSON

      schema-disagg-abort-multi-crash-test-disagg-1 on amazon2023-disagg-asan-stress

      Host: i-0c802aebdcdf1ebeb
      Project: wiredtiger-disagg
      Commit: 74b012e6
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      schema-disagg-abort-multi-crash-test-disagg-1 task_log

      Logs:

      Task logger initialized (agent version '2026-08-28' from Evergreen build revision '6f6d33950a789b6ea8872e276f869e979098590a').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_crash_test_disagg_1_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 2.344244ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 38.833µs.
      Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
      mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 not installed, downloading...
      Autodetected Amazon Linux 2023 (OS_ID: amazon2023) on aarch64
      Autodetected DOWNLOAD_URL: https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/latest
      Overriding REVISION_ID, will download toolchain for 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542
      Ensuring /opt/mongodbtoolchain/revisions exists and has correct ownership...
      Downloading toolchain to /data/mci/0534ef80d03b88a334f4b193e5254d09/tmp/tmp.yi98IjZkEz/mongodbtoolchain.tar.gz from https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/mongodbtoolchain-93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542.tar.gz
      Getting toolchain revision from archive...
      Checking downloaded archive integrity...
      Extracting toolchain revision 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 from archive...
      Running post install script...
      Existing v2 found, uninstalling...
      Existing v3 found, uninstalling...
      Existing v4 found, uninstalling...
      Existing v5 found, uninstalling...
      Installing v2...
      Installing v3...
      Installing v4...
      Installing v5...
      Cleaning up...
      Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3m5.310922499s.
      Finished running pre-task commands in 3m5.313599555s.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/74b012e60d0de08efef42edbcbcf8489c79b5661/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 7.299243954s.
      Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
        [ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
        export TESTUTIL_TSAN=1
      elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 0 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3471
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_crash_test_disagg_1_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-128-25-249.ec2.internal
      INVOCATION_ID=7d8d9cc27b65402292890791630d3b97
      JASPER_ID=b2198169-7bc1-44ec-aab7-fc1251cee3ab
      JASPER_MANAGER=f8023671-eaa8-41b1-9d88-9c0004d3f21b
      JOURNAL_STREAM=8:16793
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      LOGNAME=ec2-user
      LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/test/evergreen/asan_leaks.supp
      MAIL=/var/spool/mail/ec2-user
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      PWD=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3387
      S_COLORS=auto
      TEMP=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
      TESTUTIL_BYPASS_ASAN=1
      TMP=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
      TMPDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      echo "Dump Environment"
      printenv | sort
      build_dir=$(cd ../../../ && pwd)
      for i in $(seq 10); do
        echo "Iteration $i/10"
        ./test_schema_disagg_abort -b "$build_dir" -r lf -k l100 -k f200 -T 12 -u 64 -t 300
      done
      Iteration 1/10
      Running test command: ./test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -r lf -k l100 -k f200 -T 12 -u 64 -t 300
      Parent: roles lf; 12 schema threads; pool 64 slots; switch every 0s; kill leader@100 follower@200 lone@0; stop at 300s
      CONFIG: test_schema_disagg_abort -r lf -k f200 -u 64 -T 12 -t 300 -PSD14895861,E9323711
      Running test command: /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 1 -A f -h WT_TEST.test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -T 12 -u 64 -R 5 -W 4 -PSD14895861,E9323711
      Node 1: starting as follower
      Running test command: /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A l -h WT_TEST.test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -T 12 -u 64 -R 3 -W 6 -PSD14895861,E9323711
      Node 0: starting as leader
      Node 0: checkpoint 1 complete; stable timestamp = 3
      Node 0: checkpoint 2 complete; stable timestamp = 8
      Node 0: checkpoint 3 complete; stable timestamp = 16
      Node 0: checkpoint 4 complete; stable timestamp = 54
      Node 0: checkpoint 5 complete; stable timestamp = 97
      Node 0: checkpoint 6 complete; stable timestamp = 151
      Node 0: checkpoint 7 complete; stable timestamp = 197
      Node 0: checkpoint 8 complete; stable timestamp = 244
      Node 0: checkpoint 9 complete; stable timestamp = 295
      Node 0: checkpoint 10 complete; stable timestamp = 323
      Node 0: checkpoint 11 complete; stable timestamp = 430
      Node 0: checkpoint 12 complete; stable timestamp = 533
      Node 0: checkpoint 13 complete; stable timestamp = 650
      Node 0: checkpoint 14 complete; stable timestamp = 796
      

      logs

      schema-disagg-abort-multi-crash-test-disagg-1 task_log

      Logs:

      [1787941824:412140][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_meta_checkpoint_last_name(WT_SESSION_IMPL *, const char *, const char **, int64_t *, uint64_t *), 211: Error at src/meta/meta_ckpt.c:211: "__wt_ckpt_last_name(session, config, namep, orderp, timep)" failed: WT_NOTFOUND: item not found
      [1787941824:412184][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __btree_pin_hs_dhandle(WT_SESSION_IMPL *, WT_BTREE *), 66: Error at src/btree/bt_handle.c:66: "__wt_meta_checkpoint_last_name(session, WT_HS_URI_SHARED, &hs_checkpoint_name, NULL, NULL)" failed: WT_NOTFOUND: item not found
      [1787941824:412193][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412207][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412213][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412218][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412223][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412227][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412232][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1787941824:412237][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412241][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412246][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412251][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412255][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412260][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412264][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412269][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1787941824:412274][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412278][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_meta_checkpoint_last_name(WT_SESSION_IMPL *, const char *, const char **, int64_t *, uint64_t *), 211: Error at src/meta/meta_ckpt.c:211: "__wt_ckpt_last_name(session, config, namep, orderp, timep)" failed: WT_NOTFOUND: item not found
      [1787941824:412284][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __btree_pin_hs_dhandle(WT_SESSION_IMPL *, WT_BTREE *), 66: Error at src/btree/bt_handle.c:66: "__wt_meta_checkpoint_last_name(session, WT_HS_URI_SHARED, &hs_checkpoint_name, NULL, NULL)" failed: WT_NOTFOUND: item not found
      [1787941824:412288][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412293][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412298][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412303][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412307][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412312][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412316][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1787941824:412321][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412326][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412331][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412336][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412340][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412345][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found
      [1787941824:412349][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoi
      

      logs

      schema-disagg-abort-multi-crash-test-disagg-1 task_log

      Logs:

      test_schema_disagg_abort: FAILED: node1 terminated unexpectedly: signal 6: Invalid argument
      

      logs

      schema-disagg-abort-multi-crash-test-disagg-1 task_log

      Logs:

      Running task commands failed: running command: command failed: shell script encountered problem: exit code 134
      Finished running task commands in 1m7.968495877s.
      Task completed - FAILURE.
      Running post-task commands.
      

      logs

      schema-disagg-abort-multi-crash-test-disagg-1 task_log

      Logs:

      #0  0x0000ffff9b4c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffff9b4c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffff9b47b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffff9b462224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffff9c102fac [PAC] in __wt_abort (session=0x0) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000aaaac6ae3ce8 in testutil_die (e=22, fmt=0xaaaac6afaec0 <str> "%s terminated unexpectedly: %s %d") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/utility/misc.c:75
      #5  0x0000aaaac6ad4b98 in die_on_child_status (proc=0xaaaac6ba1238 <run_children[children]+24>, status=SUBPROC_KILLED, code=6) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:140
      #6  0x0000aaaac6ad42e0 in child_alive (proc=0xaaaac6ba1238 <run_children[children]+24>) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:157
      #7  0x0000aaaac6ad40b8 in children_poll (children=0xaaaac6ba1220 <run_children[children]>) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:171
      #8  0x0000aaaac6ad1c14 in run_children (cfg=0xffff99a00020, self_path=0xffff99a02190 "/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:313
      #9  0x0000aaaac6ad10e0 in parent_main (cfg=0xffff99a00020, self_path=0xffff99a02190 "/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:407
      #10 0x0000aaaac6acaa6c in main (argc=15, argv=0xffffe3842228) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/main.c:461
      #0  0x0000ffffb38c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #0  0x0000ffffb38c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
      #1  0x0000ffffb387b320 [PAC] in raise () from /lib64/libc.so.6
      #2  0x0000ffffb3862224 [PAC] in abort () from /lib64/libc.so.6
      #3  0x0000ffffb4502fac [PAC] in __wt_abort (session=0xffffb2eaca40) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/os_common/os_abort.c:32
      #4  0x0000ffffb47eb900 in __wt_panic_func (session=0xffffb2eaca40, error=22, func=0xffffb4a9eb00 <__PRETTY_FUNCTION__.__disagg_check_meta_field> "int __disagg_check_meta_field(WT_SESSION_IMPL *, const char *, const char *, const WT_CONFIG_ITEM *, WT_CONFIG_ITEM *, WT_CONFIG_ITEM *)", line=446, category=WT_VERB_DEFAULT, fmt=0xffffb4a9ed40 <str> "checkpoint pickup metadata mismatch for \"%s\": the value of \"%s%.*s\" differs between the local (\"%.*s\") and the shared (\"%.*s\") metadata") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/support/err.c:633
      #5  0x0000ffffb412cf38 in __disagg_check_meta_field (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", prefix=0xffffb4a9e6e0 <str> "", key=0xffffad9d3ca0, md_cval=0xffffad9d3c60, sh_cval=0xffffad9d3ce0) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:442
      #6  0x0000ffffb412c18c in __disagg_check_meta_fields (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", prefix=0xffffb4a9e6e0 <str> "", md_parser=0xffffad9d3a20, sh_parser=0xffffad9d3a70) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:479
      #7  0x0000ffffb412bd20 in __disagg_check_meta_all_fields (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", md_value=0x51a000c0cc80 "access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_m"..., sh_value=0x52a000038557 "access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_m"...) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:522
      #8  0x0000ffffb4129130 in __disagg_check_meta_match (session=0xffffb2eaca40, sh_cursor=0x51a000c06680, md_cursor=0x51a000c03680) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:593
      #9  0x0000ffffb41234ac in __disagg_apply_checkpoint_meta (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, ckpt_schema_epoch=955, is_startup=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:941
      #10 0x0000ffffb411f09c in __disagg_adopt_checkpoint_meta (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, metadata=0xffffadd1f020, is_startup=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1634
      #11 0x0000ffffb411ca30 in __disagg_pick_up_checkpoint (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, superseded_ok=true) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1778
      #12 0x0000ffffb4118970 in __disagg_pick_up_checkpoint_meta (session=0xffffb2e9c840, meta_data=0x50b000052000 "metadata_lsn=1952,metadata_checksum=f717886b,database_size=2382193,version=2,compatible_version=2", meta_data_size=97, force=true, superseded_ok=true) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:2000
      #13 0x0000ffffb4117b5c in __disagg_adopt_deferred_checkpoint_meta (session=0xffffb2e9c840, meta_data=0x50b000052000 "metadata_lsn=1952,metadata_checksum=f717886b,database_size=2382193,version=2,compatible_version=2", meta_data_size=97) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:2051
      #14 0x0000ffffb41176d0 in __wti_disagg_deferred_pickup_retry (session=0xffffb2e9c840, force=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1536
      #15 0x0000ffffb4116f54 in __disagg_deferred_pickup_server (arg=0xffffb2e9c840) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1361
      #16 0x0000ffffb51056e0 in asan_thread_start () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239
      #17 0x0000ffffb38c2834 in start_thread () from /lib64/libc.so.6
      #18 0x0000ffffb3866e5c [PAC] in thread_start () from /lib64/libc.so.6
      

      logs

            Assignee:
            Jie Chen
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: