-
Type:
Build Failure
-
Resolution: Gone away
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: DHandles, Sweep Server
-
Storage Engines - Transactions
-
48.488
-
None
-
None
-
- schema-disagg-abort-crash-test-disagg-1
- schema-disagg-abort-follower-stepup-test-disagg-1
- schema-disagg-abort-multi-crash-test-disagg-1
- schema-disagg-abort-switch-test-disagg-1
- schema-disagg-abort-unique-crash-test-disagg-1
- schema-disagg-abort-unique-follower-stepup-test-disagg-1
- schema-disagg-abort-unique-multi-crash-test-disagg-1
- schema-disagg-abort-unique-multi-switch-crash-test-disagg-1
- schema-disagg-abort-unique-switch-test-disagg-1
-
26
schema-disagg-abort-unique-switch-test-disagg-1 on amazon2023-disagg-asan-stress
Host: i-0dd5f59ab2d90d919
Project: wiredtiger-disagg
Commit: c29c44c5
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
Task logger initialized (agent version '2026-09-14a' from Evergreen build revision '36e46733f111f43eda86416bcd53d097cafbd9c8').
Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_unique_switch_test_disagg_1_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29', execution 0.
Running pre-task commands.
Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 20.980202ms.
Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 39.661µs.
Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 not installed, downloading...
Autodetected Amazon Linux 2023 (OS_ID: amazon2023) on aarch64
Autodetected DOWNLOAD_URL: https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/latest
Overriding REVISION_ID, will download toolchain for 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542
Ensuring /opt/mongodbtoolchain/revisions exists and has correct ownership...
Downloading toolchain to /data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp/tmp.yu9orUwSZH/mongodbtoolchain.tar.gz from https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/mongodbtoolchain-93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542.tar.gz
Getting toolchain revision from archive...
Checking downloaded archive integrity...
Extracting toolchain revision 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 from archive...
Running post install script...
Existing v2 found, uninstalling...
Existing v3 found, uninstalling...
Existing v4 found, uninstalling...
Existing v5 found, uninstalling...
Installing v2...
Installing v3...
Installing v4...
Installing v5...
Cleaning up...
Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3m3.0219473s.
Finished running pre-task commands in 3m3.043270126s.
Running task commands.
Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 7.984906692s.
Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
export WT_TOPDIR=$(git rev-parse --show-toplevel)
export WT_BUILDDIR=$WT_TOPDIR/cmake_build
if [ "$OS" = "Windows_NT" ]; then
export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
else
export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
export LD_LIBRARY_PATH=$WT_BUILDDIR
fi
# Create the common sanitizer options and export the specific sanitizer environment
# variables.
COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
[ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
export TESTUTIL_BYPASS_ASAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
export TESTUTIL_MSAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
export TESTUTIL_TSAN=1
elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
export TESTUTIL_UBSAN=1
fi
if [[ 0 -eq 1 ]]; then
# The preloaded library should be downloaded/built during "configure wiredtiger" step.
# DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
# and as close as possible to the test execution, since it affects all the binaries.
export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
if [ ! -f "$LD_PRELOAD" ]; then
echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
exit 1
fi
fi
# Set extensions to load
EXT="extensions=["
EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
EXT="$EXT ]"
export EXT
export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
Dump Environment
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
BASH_FUNC_which%%=() { ( alias;
CI=true
EVR_AGENT_PID=3618
EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_unique_switch_test_disagg_1_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29
EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
GOCACHE=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
GOTRACEBACK=none
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/ec2-user
HOSTNAME=ip-10-122-33-40.ec2.internal
INVOCATION_ID=1045e6d3c3ac40d69ab7d205d924a8e7
JASPER_ID=4f1924b9-a39f-4e44-b476-92405927a351
JASPER_MANAGER=d20e9636-2ba9-4d49-8532-a17d98e95baf
JOURNAL_STREAM=8:13123
LANG=C.UTF-8
LC_ALL=C
LD_LIBRARY_PATH=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
LESSOPEN=||/usr/bin/lesspipe.sh %s
LOGNAME=ec2-user
LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/test/evergreen/asan_leaks.supp
MAIL=/var/spool/mail/ec2-user
PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
PWD=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
SHELL=/bin/bash
SHLVL=1
SYSTEMD_COLORS=false
SYSTEMD_EXEC_PID=3455
S_COLORS=auto
TEMP=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
TESTUTIL_BYPASS_ASAN=1
TMP=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
TMPDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
USER=ec2-user
WT_BUILDDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build
WT_TOPDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger
_=/usr/bin/printenv
which_declare=declare -f
}
echo "Dump Environment"
printenv | sort
build_dir=$(cd ../../../ && pwd)
for i in $(seq 10); do
echo "Iteration $i/10"
./test_schema_disagg_abort -b "$build_dir" -r l -s 30 -q -T 12 -u 64 -t 300
Iteration 1/10
Running test command: ./test_schema_disagg_abort -b /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build -r l -s 30 -q -T 12 -u 64 -t 300
Parent: roles l; 12 schema threads; pool 64 slots; switch every 30s; kill leader@0 follower@0 lone@0; stop at 300s
CONFIG: test_schema_disagg_abort -r l -s 30 -u 64 -T 12 -t 300 -PSD10062897,E4611578
Running test command: /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A l -h WT_TEST.test_schema_disagg_abort -b /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build -T 12 -u 64 -q -s 30 -PSD10062897,E4611578
Node 0: starting as leader
Node 0: checkpoint 1 at 1 (lsn 5)
Node 0: checkpoint 2 at 7 (lsn 10)
Node 0: checkpoint 3 at 40 (lsn 54)
Node 0: checkpoint 4 at 79 (lsn 137)
Node 0: checkpoint 5 at 121 (lsn 227)
Node 0: checkpoint 6 at 142 (lsn 299)
Node 0: checkpoint 7 at 230 (lsn 443)
Node 0: checkpoint 8 at 316 (lsn 651)
Parent: directing switch 1
Node 0: checkpoint 9 at 409 (lsn 796)
Node 0: checkpoint 10 at 553 (lsn 1023)
Node 0: checkpoint 11 at 601 (lsn 1128)
Node 0: checkpoint 12 at 652 (lsn 1222)
Node 0: step-down checkpoint 13 at 806 (lsn 1375)
Node 0: step-down emitted 769 events in 12035 ms (budget 120 s)
Node 0: no peer to hand over to; continuing alone
Node 0: now follower
Node 0: switch 1 complete
Parent: directing switch 2
Node 0: now leader
Node 0: switch 2 complete
Node 0: checkpoint 1 at 3375 (lsn 4269)
Node 0: checkpoint 2 at 3576 (lsn 4725)
Node 0: checkpoint 3 at 3764 (lsn 5141)
Node 0: checkpoint 4 at 3971 (lsn 5645)
Node 0: checkpoint 5 at 4182 (lsn 6066)
Node 0: checkpoint 6 at 4238 (lsn 6215)
Node 0: checkpoint 7 at 4352 (lsn 6506)
Node 0: checkpoint 8 at 4490 (lsn 6823)
Node 0: checkpoint 9 at 4545 (lsn 6984)
Parent: directing switch 3
Node 0: checkpoint 10 at 4601 (lsn 7161)
Node 0: checkpoint 11 at 4714 (lsn 7409)
Node 0: checkpoint 12 at 4836 (lsn 7665)
Node 0: checkpoint 13 at 5003 (lsn 8016)
Node 0: checkpoint 14 at 5066 (lsn 8352)
Node 0: step-down checkpoint 15 at 5075 (lsn 8376)
Node 0: step-down emitted 1015 events in 12000 ms (budget 120 s)
Node 0: no peer to hand over to; continuing alone
Node 0: now follower
Node 0: switch 3 complete
Parent: directing switch 4
Node 0: now leader
Node 0: switch 4 complete
Node 0: checkpoint 1 at 7661 (lsn 11830)
Node 0: checkpoint 2 at 7723 (lsn 12098)
Node 0: checkpoint 3 at 7979 (lsn 13207)
Node 0: checkpoint 4 at 8185 (lsn 13795)
Node 0: checkpoint 5 at 8263 (lsn 14040)
Node 0: checkpoint 6 at 8472 (lsn 14418)
Node 0: checkpoint 7 at 8618
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
==17463==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0xffffb6f9d758 bp 0xffffa8925460 sp 0xffffa8925060 T10)
==17463==The signal is caused by a READ memory access.
==17463==Hint: address points to the zero page.
#0 0xffffb6f9d758 in __sweep_expire /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238:5
#1 0xffffb6f9abf8 in __sweep_server /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:561:13
#2 0xffffb7f056dc in asan_thread_start(void*) /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
#3 0xffffb66c2830 in start_thread (/lib64/libc.so.6+0x90830) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
#4 0xffffb6666e58 in thread_start (/lib64/libc.so.6+0x34e58) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
AddressSanitizer can not provide additional info.
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
SUMMARY: AddressSanitizer: SEGV /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238:5 in __sweep_expire
Thread T10 created by T0 here:
#0 0xffffb7eecb74 in pthread_create /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
#1 0xffffb732ef94 in __wt_thread_create /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/os_posix/os_thread.c:71:5
#2 0xffffb6f9a37c in __wti_sweep_create /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:664:5
#3 0xffffb6f7f184 in __wti_connection_workers /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_open.c:316:5
#4 0xffffb6eb7c24 in wiredtiger_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_api.c:3836:5
#5 0xaaaabd4c7304 in testutil_wiredtiger_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/utility/misc.c:406:5
#6 0xaaaabd4aef2c in node_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/node.c:122:5
#7 0xaaaabd4aea98 in node_main /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/node.c:454:5
#8 0xaaaabd4aaf8c in main /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/main.c:450:17
#9 0xffffb6667a74 in __libc_start_call_main (/lib64/libc.so.6+0x35a74) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
#10 0xaffffb6667b58 (<unknown module>)
#11 0x2eaaaabd4a19ec (<unknown module>)
==17463==ABORTING
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
test_schema_disagg_abort: FAILED: node0 terminated unexpectedly: signal 6: Invalid argument
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
Running task commands failed: running command: command failed: shell script encountered problem: exit code 134 Finished running task commands in 1h8m0.961166712s. Task completed - FAILURE. Running post-task commands.
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
#0 0x0000ffff91cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffff91cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffff91c7b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffff91c62224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffff929033f0 [PAC] in __wt_abort (session=0x0) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/os_common/os_abort.c:32 #4 0x0000aaaace083edc in testutil_die (e=22, fmt=0xaaaace09b140 <str> "%s terminated unexpectedly: %s %d") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/utility/misc.c:75 #5 0x0000aaaace0751cc in die_on_child_status (proc=0xaaaace1411c0 <run_children[children]>, status=SUBPROC_KILLED, code=6) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:140 #6 0x0000aaaace074914 in child_alive (proc=0xaaaace1411c0 <run_children[children]>) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:157 #7 0x0000aaaace0746ec in children_poll (children=0xaaaace1411c0 <run_children[children]>) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:171 #8 0x0000aaaace072248 in run_children (cfg=0xffff90200020, self_path=0xffff90202190 "/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:313 #9 0x0000aaaace071714 in parent_main (cfg=0xffff90200020, self_path=0xffff90202190 "/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:407 #10 0x0000aaaace06b044 in main (argc=14, argv=0xffffdd9fa008) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/main.c:461 #0 0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #0 0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x0000ffffb667b320 [PAC] in raise () from /lib64/libc.so.6 #2 0x0000ffffb6662224 [PAC] in abort () from /lib64/libc.so.6 #3 0x0000ffffb7e5127c [PAC] in Abort () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163 #4 0x0000ffffb7e4f164 in __sanitizer::Die() () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58 #5 0x0000ffffb7f0cac0 in ~ScopedInErrorReport () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:193 #6 0x0000ffffb7f0c418 in ReportDeadlySignal () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:221 #7 0x0000ffffb7f0b94c in AsanOnDeadlySignal () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_posix.cpp:40 #8 <signal handler called> #9 0x0000ffffb6f9d758 in __sweep_expire (session=0xffffb5e128c0, now=1789432419) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238 #10 0x0000ffffb6f9abfc in __sweep_server (arg=0xffffb5e128c0) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:561 #11 0x0000ffffb7f056e0 in asan_thread_start () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239 #12 0x0000ffffb66c2834 in start_thread () from /lib64/libc.so.6 #13 0x0000ffffb6666e5c [PAC] in thread_start () from /lib64/libc.so.6
schema-disagg-abort-unique-switch-test-disagg-1 task_log
Logs:
Finished command 'shell.exec' in function 'dump stacktraces' (step 1 of 9) in block 'post' in 2.852865817s.
Running command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post'.
Putting files matching filter '[wiredtiger/cmake_build/*stacktrace.txt]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
Finished command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post' in 196.457792ms.
Running command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post'.
Putting files matching filter '[wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
File filter 'wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG' matched no files.
Finished command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post' in 52.09099ms.
Running command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post'.
Putting files matching filter '[wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
File filter 'wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload' matched no files.
Finished command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post' in 46.840821ms.
Running command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post'.
if [ -d "WT_TEST" ]; then
# Dump stderr/stdout contents generated by the C libraries onto console for Python tests
find "WT_TEST" -name "std*.txt" ! -empty -exec bash -c "echo 'Contents from {}:'; cat '{}'" \;
fi
Finished command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post' in 1.383234ms.
Running command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post'.
mkdir stat_files
python3 ../test/evergreen/collect_stat_files.py -d stat_files
Finished command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post' in 57.728629ms.
Running command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post'.
Beginning to build archive.
[message='successfully created archive' num_files='4' target='/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/stat_files.tar.gz']
Finished command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' in 29.543771ms.
Running command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post'.
Putting local file 'wiredtiger/cmake_build/stat_files.tar.gz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz).
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz' in bucket 'build_external' (attempt 1 of 5).
Finished command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post' in 370.35074ms.
Running command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post'.
Beginning to build archive.
[message='successfully created archive' num_files='8595' target='/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger.tgz']
Finished command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' in 1.662733423s.
Running command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post'.
Putting local file 'wiredtiger.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz).
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz' in bucket 'build_external' (attempt 1 of 5).
Finished command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post' in 1.434121765s.
Running command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post'.
Beginning to build archive.
No files were archived.
Finished command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' in 491.922µs.
Running command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post'.
Putting local file 'wt-hang-analyzer.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz).
Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' in bucket 'build_external' (attempt 1 of 5).
File '/data/mci/83c1a8aa392bffbef0a877d0d1152470/wt-hang-analyzer.tgz' not found and skip missing is true, exiting without error.
Finished command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post' in 14.105694ms.
Running command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post'.
Finished command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post' in 502.273724ms.
Finished running post-task commands in 7.222493456s.
reading test log spec; falling back to default spec: open /data/mci/83c1a8aa392bffbef0a877d0d1152470/build/TestLogs/log_spec.yaml: no such file or directory