-
Type:
Build Failure
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Schema Management
-
Storage Engines, Storage Engines - Foundations
-
407.643
-
None
-
None
-
55
schema-disagg-abort-multi-crash-test-disagg-1 on amazon2023-disagg-asan-stress
Host: i-0c802aebdcdf1ebeb
Project: wiredtiger-disagg
Commit: 74b012e6
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
schema-disagg-abort-multi-crash-test-disagg-1 task_log
Logs:
Task logger initialized (agent version '2026-08-28' from Evergreen build revision '6f6d33950a789b6ea8872e276f869e979098590a').
Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_crash_test_disagg_1_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57', execution 0.
Running pre-task commands.
Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 2.344244ms.
Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 38.833µs.
Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 not installed, downloading...
Autodetected Amazon Linux 2023 (OS_ID: amazon2023) on aarch64
Autodetected DOWNLOAD_URL: https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/latest
Overriding REVISION_ID, will download toolchain for 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542
Ensuring /opt/mongodbtoolchain/revisions exists and has correct ownership...
Downloading toolchain to /data/mci/0534ef80d03b88a334f4b193e5254d09/tmp/tmp.yi98IjZkEz/mongodbtoolchain.tar.gz from https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/mongodbtoolchain-93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542.tar.gz
Getting toolchain revision from archive...
Checking downloaded archive integrity...
Extracting toolchain revision 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 from archive...
Running post install script...
Existing v2 found, uninstalling...
Existing v3 found, uninstalling...
Existing v4 found, uninstalling...
Existing v5 found, uninstalling...
Installing v2...
Installing v3...
Installing v4...
Installing v5...
Cleaning up...
Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3m5.310922499s.
Finished running pre-task commands in 3m5.313599555s.
Running task commands.
Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/74b012e60d0de08efef42edbcbcf8489c79b5661/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 7.299243954s.
Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
export WT_TOPDIR=$(git rev-parse --show-toplevel)
export WT_BUILDDIR=$WT_TOPDIR/cmake_build
if [ "$OS" = "Windows_NT" ]; then
export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
else
export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
export LD_LIBRARY_PATH=$WT_BUILDDIR
fi
# Create the common sanitizer options and export the specific sanitizer environment
# variables.
COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
[ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
export TESTUTIL_BYPASS_ASAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
export TESTUTIL_MSAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
export TESTUTIL_TSAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
export TESTUTIL_UBSAN=1
fi
if [[ 0 -eq 1 ]]; then
# The preloaded library should be downloaded/built during "configure wiredtiger" step.
# DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
# and as close as possible to the test execution, since it affects all the binaries.
export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
if [ ! -f "$LD_PRELOAD" ]; then
echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
exit 1
fi
fi
# Set extensions to load
EXT="extensions=["
EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
EXT="$EXT ]"
export EXT
export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
Dump Environment
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
BASH_FUNC_which%%=() { ( alias;
CI=true
EVR_AGENT_PID=3471
EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_multi_crash_test_disagg_1_74b012e60d0de08efef42edbcbcf8489c79b5661_26_08_28_16_52_57
EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
GOCACHE=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
GOTRACEBACK=none
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/ec2-user
HOSTNAME=ip-10-128-25-249.ec2.internal
INVOCATION_ID=7d8d9cc27b65402292890791630d3b97
JASPER_ID=b2198169-7bc1-44ec-aab7-fc1251cee3ab
JASPER_MANAGER=f8023671-eaa8-41b1-9d88-9c0004d3f21b
JOURNAL_STREAM=8:16793
LANG=C.UTF-8
LC_ALL=C
LD_LIBRARY_PATH=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
LESSOPEN=||/usr/bin/lesspipe.sh %s
LOGNAME=ec2-user
LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/test/evergreen/asan_leaks.supp
MAIL=/var/spool/mail/ec2-user
PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
PWD=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
SHELL=/bin/bash
SHLVL=1
SYSTEMD_COLORS=false
SYSTEMD_EXEC_PID=3387
S_COLORS=auto
TEMP=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
TESTUTIL_BYPASS_ASAN=1
TMP=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
TMPDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/tmp
USER=ec2-user
WT_BUILDDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build
WT_TOPDIR=/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger
_=/usr/bin/printenv
which_declare=declare -f
}
echo "Dump Environment"
printenv | sort
build_dir=$(cd ../../../ && pwd)
for i in $(seq 10); do
echo "Iteration $i/10"
./test_schema_disagg_abort -b "$build_dir" -r lf -k l100 -k f200 -T 12 -u 64 -t 300
done
Iteration 1/10
Running test command: ./test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -r lf -k l100 -k f200 -T 12 -u 64 -t 300
Parent: roles lf; 12 schema threads; pool 64 slots; switch every 0s; kill leader@100 follower@200 lone@0; stop at 300s
CONFIG: test_schema_disagg_abort -r lf -k f200 -u 64 -T 12 -t 300 -PSD14895861,E9323711
Running test command: /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 1 -A f -h WT_TEST.test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -T 12 -u 64 -R 5 -W 4 -PSD14895861,E9323711
Node 1: starting as follower
Running test command: /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A l -h WT_TEST.test_schema_disagg_abort -b /data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build -T 12 -u 64 -R 3 -W 6 -PSD14895861,E9323711
Node 0: starting as leader
Node 0: checkpoint 1 complete; stable timestamp = 3
Node 0: checkpoint 2 complete; stable timestamp = 8
Node 0: checkpoint 3 complete; stable timestamp = 16
Node 0: checkpoint 4 complete; stable timestamp = 54
Node 0: checkpoint 5 complete; stable timestamp = 97
Node 0: checkpoint 6 complete; stable timestamp = 151
Node 0: checkpoint 7 complete; stable timestamp = 197
Node 0: checkpoint 8 complete; stable timestamp = 244
Node 0: checkpoint 9 complete; stable timestamp = 295
Node 0: checkpoint 10 complete; stable timestamp = 323
Node 0: checkpoint 11 complete; stable timestamp = 430
Node 0: checkpoint 12 complete; stable timestamp = 533
Node 0: checkpoint 13 complete; stable timestamp = 650
Node 0: checkpoint 14 complete; stable timestamp = 796
schema-disagg-abort-multi-crash-test-disagg-1 task_log
Logs:
[1787941824:412140][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_meta_checkpoint_last_name(WT_SESSION_IMPL *, const char *, const char **, int64_t *, uint64_t *), 211: Error at src/meta/meta_ckpt.c:211: "__wt_ckpt_last_name(session, config, namep, orderp, timep)" failed: WT_NOTFOUND: item not found [1787941824:412184][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __btree_pin_hs_dhandle(WT_SESSION_IMPL *, WT_BTREE *), 66: Error at src/btree/bt_handle.c:66: "__wt_meta_checkpoint_last_name(session, WT_HS_URI_SHARED, &hs_checkpoint_name, NULL, NULL)" failed: WT_NOTFOUND: item not found [1787941824:412193][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412207][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412213][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412218][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412223][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412227][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412232][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1787941824:412237][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412241][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412246][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412251][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412255][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412260][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412264][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412269][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1787941824:412274][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412278][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_meta_checkpoint_last_name(WT_SESSION_IMPL *, const char *, const char **, int64_t *, uint64_t *), 211: Error at src/meta/meta_ckpt.c:211: "__wt_ckpt_last_name(session, config, namep, orderp, timep)" failed: WT_NOTFOUND: item not found [1787941824:412284][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __btree_pin_hs_dhandle(WT_SESSION_IMPL *, WT_BTREE *), 66: Error at src/btree/bt_handle.c:66: "__wt_meta_checkpoint_last_name(session, WT_HS_URI_SHARED, &hs_checkpoint_name, NULL, NULL)" failed: WT_NOTFOUND: item not found [1787941824:412288][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412293][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412298][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412303][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412307][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 389: Error at src/btree/bt_curnext.c:389: "ret" failed: WT_NOTFOUND: item not found [1787941824:412312][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412316][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __wt_btcur_next(WT_CURSOR_BTREE *, _Bool), 804: Error at src/btree/bt_curnext.c:804: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1787941824:412321][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412326][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412331][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412336][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412340][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __curfile_next(WT_CURSOR *), 192: Error at src/cursor/cur_file.c:192: "ret" failed: WT_NOTFOUND: item not found [1787941824:412345][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoint.15, checkpoint-pick-up: [WT_VERB_ERROR_RETURNS][ERROR]: int __cursor_row_next(WT_CURSOR_BTREE *, _Bool, _Bool, size_t *, _Bool *), 334: Error at src/btree/bt_curnext.c:334: "ret" failed: WT_NOTFOUND: item not found [1787941824:412349][13484:0xffffae8e50e0], file:WiredTigerShared.wt_stable/WiredTigerCheckpoi
schema-disagg-abort-multi-crash-test-disagg-1 task_log
Logs:
test_schema_disagg_abort: FAILED: node1 terminated unexpectedly: signal 6: Invalid argument
schema-disagg-abort-multi-crash-test-disagg-1 task_log
Logs:
Running task commands failed: running command: command failed: shell script encountered problem: exit code 134 Finished running task commands in 1m7.968495877s. Task completed - FAILURE. Running post-task commands.
schema-disagg-abort-multi-crash-test-disagg-1 task_log
Logs:
#0 0x0000ffff9b4c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff9b4c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff9b47b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff9b462224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff9c102fac [PAC] in __wt_abort (session=0x0) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000aaaac6ae3ce8 in testutil_die (e=22, fmt=0xaaaac6afaec0 <str> "%s terminated unexpectedly: %s %d") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/utility/misc.c:75 #5 0x0000aaaac6ad4b98 in die_on_child_status (proc=0xaaaac6ba1238 <run_children[children]+24>, status=SUBPROC_KILLED, code=6) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:140 #6 0x0000aaaac6ad42e0 in child_alive (proc=0xaaaac6ba1238 <run_children[children]+24>) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:157 #7 0x0000aaaac6ad40b8 in children_poll (children=0xaaaac6ba1220 <run_children[children]>) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:171 #8 0x0000aaaac6ad1c14 in run_children (cfg=0xffff99a00020, self_path=0xffff99a02190 "/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:313 #9 0x0000aaaac6ad10e0 in parent_main (cfg=0xffff99a00020, self_path=0xffff99a02190 "/data/mci/0534ef80d03b88a334f4b193e5254d09/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/parent.c:407 #10 0x0000aaaac6acaa6c in main (argc=15, argv=0xffffe3842228) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/test/csuite/schema_disagg_abort/main.c:461 #0 0x0000ffffb38c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffb38c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffb387b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffb3862224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffffb4502fac [PAC] in __wt_abort (session=0xffffb2eaca40) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000ffffb47eb900 in __wt_panic_func (session=0xffffb2eaca40, error=22, func=0xffffb4a9eb00 <__PRETTY_FUNCTION__.__disagg_check_meta_field> "int __disagg_check_meta_field(WT_SESSION_IMPL *, const char *, const char *, const WT_CONFIG_ITEM *, WT_CONFIG_ITEM *, WT_CONFIG_ITEM *)", line=446, category=WT_VERB_DEFAULT, fmt=0xffffb4a9ed40 <str> "checkpoint pickup metadata mismatch for \"%s\": the value of \"%s%.*s\" differs between the local (\"%.*s\") and the shared (\"%.*s\") metadata") at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/support/err.c:633 #5 0x0000ffffb412cf38 in __disagg_check_meta_field (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", prefix=0xffffb4a9e6e0 <str> "", key=0xffffad9d3ca0, md_cval=0xffffad9d3c60, sh_cval=0xffffad9d3ce0) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:442 #6 0x0000ffffb412c18c in __disagg_check_meta_fields (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", prefix=0xffffb4a9e6e0 <str> "", md_parser=0xffffad9d3a20, sh_parser=0xffffad9d3a70) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:479 #7 0x0000ffffb412bd20 in __disagg_check_meta_all_fields (session=0xffffb2eaca40, uri=0x52a000038534 "file:schema_0_11_50_0.wt_stable", md_value=0x51a000c0cc80 "access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_m"..., sh_value=0x52a000038557 "access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_m"...) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:522 #8 0x0000ffffb4129130 in __disagg_check_meta_match (session=0xffffb2eaca40, sh_cursor=0x51a000c06680, md_cursor=0x51a000c03680) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:593 #9 0x0000ffffb41234ac in __disagg_apply_checkpoint_meta (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, ckpt_schema_epoch=955, is_startup=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:941 #10 0x0000ffffb411f09c in __disagg_adopt_checkpoint_meta (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, metadata=0xffffadd1f020, is_startup=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1634 #11 0x0000ffffb411ca30 in __disagg_pick_up_checkpoint (session=0xffffb2eaca40, ckpt_meta=0xffffad83fa60, superseded_ok=true) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1778 #12 0x0000ffffb4118970 in __disagg_pick_up_checkpoint_meta (session=0xffffb2e9c840, meta_data=0x50b000052000 "metadata_lsn=1952,metadata_checksum=f717886b,database_size=2382193,version=2,compatible_version=2", meta_data_size=97, force=true, superseded_ok=true) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:2000 #13 0x0000ffffb4117b5c in __disagg_adopt_deferred_checkpoint_meta (session=0xffffb2e9c840, meta_data=0x50b000052000 "metadata_lsn=1952,metadata_checksum=f717886b,database_size=2382193,version=2,compatible_version=2", meta_data_size=97) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:2051 #14 0x0000ffffb41176d0 in __wti_disagg_deferred_pickup_retry (session=0xffffb2e9c840, force=false) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1536 #15 0x0000ffffb4116f54 in __disagg_deferred_pickup_server (arg=0xffffb2e9c840) at /data/mci/7db885bdebb8cdb024aef229401e06ad/wiredtiger/src/conn/conn_layered_checkpoint_pick_up.c:1361 #16 0x0000ffffb51056e0 in asan_thread_start () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239 #17 0x0000ffffb38c2834 in start_thread () from /lib64/libc.so.6 #18 0x0000ffffb3866e5c [PAC] in thread_start () from /lib64/libc.so.6
- duplicates
-
WT-18068 Follower reads stale btree ID after leader create/drop/create of a layered table
-
- In Progress
-