failed: schema-disagg-abort-unique-switch-test-disagg-1 on amazon2023-disagg-asan-stress [wiredtiger-disagg @ c29c44c5]

XMLWordPrintableJSON

    schema-disagg-abort-unique-switch-test-disagg-1 on amazon2023-disagg-asan-stress

    Host: i-0dd5f59ab2d90d919
    Project: wiredtiger-disagg
    Commit: c29c44c5
    Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

    Task Logs:

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    Task logger initialized (agent version '2026-09-14a' from Evergreen build revision '36e46733f111f43eda86416bcd53d097cafbd9c8').
    Starting task 'wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_unique_switch_test_disagg_1_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29', execution 0.
    Running pre-task commands.
    Running command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre'.
    Finished command 'shell.exec' in function 'cleanup' (step 1 of 3) in block 'pre' in 20.980202ms.
    Running command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre'.
    Finished command 'expansions.update' in function 'setup environment' (step 2 of 3) in block 'pre' in 39.661µs.
    Running command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre'.
    mongodbtoolchain 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 not installed, downloading...
    Autodetected Amazon Linux 2023 (OS_ID: amazon2023) on aarch64
    Autodetected DOWNLOAD_URL: https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/latest
    Overriding REVISION_ID, will download toolchain for 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542
    Ensuring /opt/mongodbtoolchain/revisions exists and has correct ownership...
    Downloading toolchain to /data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp/tmp.yu9orUwSZH/mongodbtoolchain.tar.gz from https://mongodbtoolchain.build.10gen.cc/toolchain/amazon2023-arm64/x86_64/mongodbtoolchain-93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542.tar.gz
    Getting toolchain revision from archive...
    Checking downloaded archive integrity...
    Extracting toolchain revision 93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542 from archive...
    Running post install script...
    Existing v2 found, uninstalling...
    Existing v3 found, uninstalling...
    Existing v4 found, uninstalling...
    Existing v5 found, uninstalling...
    Installing v2...
    Installing v3...
    Installing v4...
    Installing v5...
    Cleaning up...
    Finished command 'shell.exec' in function 'pin mongo toolchain' (step 3 of 3) in block 'pre' in 3m3.0219473s.
    Finished running pre-task commands in 3m3.043270126s.
    Running task commands.
    Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
    Fetching remote file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
    Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 7.984906692s.
    Running command 'shell.exec' in function 'schema disagg abort' (step 2 of 2).
    export WT_TOPDIR=$(git rev-parse --show-toplevel)
    export WT_BUILDDIR=$WT_TOPDIR/cmake_build
    if [ "$OS" = "Windows_NT" ]; then
      export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
    else
      export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
      export LD_LIBRARY_PATH=$WT_BUILDDIR
    fi
    # Create the common sanitizer options and export the specific sanitizer environment
    # variables.
    COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
    if [[ "-DCMAKE_BUILD_TYPE=ASan" =~ ASan ]]; then
      export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
      asan_lib=$(clang -print-file-name=libclang_rt.asan.so)
      [ -f "$asan_lib" ] || { echo "ERROR: libclang_rt.asan.so not found by clang"; exit 1; }
      export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname "$asan_lib")"
      export TESTUTIL_BYPASS_ASAN=1
    elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ MSan ]]; then
      export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
      export TESTUTIL_MSAN=1
    elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ TSan ]]; then
      export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3:history_size=7:print_suppressions=0:suppressions=$WT_TOPDIR/test/evergreen/tsan_warnings.supp"
      export TESTUTIL_TSAN=1
    elif [[ "-DCMAKE_BUILD_TYPE=ASan" =~ UBSan ]]; then
      export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
      export TESTUTIL_UBSAN=1
    fi
    if [[ 0 -eq 1 ]]; then
      # The preloaded library should be downloaded/built during "configure wiredtiger" step.
      # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
      # and as close as possible to the test execution, since it affects all the binaries.
      export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
      if [ ! -f "$LD_PRELOAD" ]; then
        echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
        exit 1
      fi
    fi
    # Set extensions to load
    EXT="extensions=["
    EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
    EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
    EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
    EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
    EXT="$EXT ]"
    export EXT
    export LSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_suppressions=0:suppressions=$(git rev-parse --show-toplevel)/test/evergreen/asan_leaks.supp"
    Dump Environment
     eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
    ASAN_OPTIONS=abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1
    BASH_FUNC_which%%=() {  ( alias;
    CI=true
    EVR_AGENT_PID=3618
    EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_asan_stress_schema_disagg_abort_unique_switch_test_disagg_1_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29
    EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
    GOCACHE=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/.gocache
    GOTRACEBACK=none
    HISTCONTROL=ignoredups
    HISTSIZE=1000
    HOME=/home/ec2-user
    HOSTNAME=ip-10-122-33-40.ec2.internal
    INVOCATION_ID=1045e6d3c3ac40d69ab7d205d924a8e7
    JASPER_ID=4f1924b9-a39f-4e44-b476-92405927a351
    JASPER_MANAGER=d20e9636-2ba9-4d49-8532-a17d98e95baf
    JOURNAL_STREAM=8:13123
    LANG=C.UTF-8
    LC_ALL=C
    LD_LIBRARY_PATH=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build:/opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/stow/llvm-v5.1VC/lib/clang/19/lib/aarch64-mongodb-linux
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    LOGNAME=ec2-user
    LSAN_OPTIONS=abort_on_error=1:disable_coredump=0:print_suppressions=0:suppressions=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/test/evergreen/asan_leaks.supp
    MAIL=/var/spool/mail/ec2-user
    PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
    PWD=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort
    SHELL=/bin/bash
    SHLVL=1
    SYSTEMD_COLORS=false
    SYSTEMD_EXEC_PID=3455
    S_COLORS=auto
    TEMP=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
    TESTUTIL_BYPASS_ASAN=1
    TMP=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
    TMPDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/tmp
    USER=ec2-user
    WT_BUILDDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build
    WT_TOPDIR=/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger
    _=/usr/bin/printenv
    which_declare=declare -f
    }
    echo "Dump Environment"
    printenv | sort
    build_dir=$(cd ../../../ && pwd)
    for i in $(seq 10); do
      echo "Iteration $i/10"
      ./test_schema_disagg_abort -b "$build_dir" -r l -s 30 -q -T 12 -u 64 -t 300
    Iteration 1/10
    Running test command: ./test_schema_disagg_abort -b /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build -r l -s 30 -q -T 12 -u 64 -t 300
    Parent: roles l; 12 schema threads; pool 64 slots; switch every 30s; kill leader@0 follower@0 lone@0; stop at 300s
    CONFIG: test_schema_disagg_abort -r l -s 30 -u 64 -T 12 -t 300 -PSD10062897,E4611578
    Running test command: /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort -r node -i 0 -A l -h WT_TEST.test_schema_disagg_abort -b /data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build -T 12 -u 64 -q -s 30 -PSD10062897,E4611578
    Node 0: starting as leader
    Node 0: checkpoint 1 at 1 (lsn 5)
    Node 0: checkpoint 2 at 7 (lsn 10)
    Node 0: checkpoint 3 at 40 (lsn 54)
    Node 0: checkpoint 4 at 79 (lsn 137)
    Node 0: checkpoint 5 at 121 (lsn 227)
    Node 0: checkpoint 6 at 142 (lsn 299)
    Node 0: checkpoint 7 at 230 (lsn 443)
    Node 0: checkpoint 8 at 316 (lsn 651)
    Parent: directing switch 1
    Node 0: checkpoint 9 at 409 (lsn 796)
    Node 0: checkpoint 10 at 553 (lsn 1023)
    Node 0: checkpoint 11 at 601 (lsn 1128)
    Node 0: checkpoint 12 at 652 (lsn 1222)
    Node 0: step-down checkpoint 13 at 806 (lsn 1375)
    Node 0: step-down emitted 769 events in 12035 ms (budget 120 s)
    Node 0: no peer to hand over to; continuing alone
    Node 0: now follower
    Node 0: switch 1 complete
    Parent: directing switch 2
    Node 0: now leader
    Node 0: switch 2 complete
    Node 0: checkpoint 1 at 3375 (lsn 4269)
    Node 0: checkpoint 2 at 3576 (lsn 4725)
    Node 0: checkpoint 3 at 3764 (lsn 5141)
    Node 0: checkpoint 4 at 3971 (lsn 5645)
    Node 0: checkpoint 5 at 4182 (lsn 6066)
    Node 0: checkpoint 6 at 4238 (lsn 6215)
    Node 0: checkpoint 7 at 4352 (lsn 6506)
    Node 0: checkpoint 8 at 4490 (lsn 6823)
    Node 0: checkpoint 9 at 4545 (lsn 6984)
    Parent: directing switch 3
    Node 0: checkpoint 10 at 4601 (lsn 7161)
    Node 0: checkpoint 11 at 4714 (lsn 7409)
    Node 0: checkpoint 12 at 4836 (lsn 7665)
    Node 0: checkpoint 13 at 5003 (lsn 8016)
    Node 0: checkpoint 14 at 5066 (lsn 8352)
    Node 0: step-down checkpoint 15 at 5075 (lsn 8376)
    Node 0: step-down emitted 1015 events in 12000 ms (budget 120 s)
    Node 0: no peer to hand over to; continuing alone
    Node 0: now follower
    Node 0: switch 3 complete
    Parent: directing switch 4
    Node 0: now leader
    Node 0: switch 4 complete
    Node 0: checkpoint 1 at 7661 (lsn 11830)
    Node 0: checkpoint 2 at 7723 (lsn 12098)
    Node 0: checkpoint 3 at 7979 (lsn 13207)
    Node 0: checkpoint 4 at 8185 (lsn 13795)
    Node 0: checkpoint 5 at 8263 (lsn 14040)
    Node 0: checkpoint 6 at 8472 (lsn 14418)
    Node 0: checkpoint 7 at 8618 
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    ==17463==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000008 (pc 0xffffb6f9d758 bp 0xffffa8925460 sp 0xffffa8925060 T10)
    ==17463==The signal is caused by a READ memory access.
    ==17463==Hint: address points to the zero page.
        #0 0xffffb6f9d758 in __sweep_expire /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238:5
        #1 0xffffb6f9abf8 in __sweep_server /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:561:13
        #2 0xffffb7f056dc in asan_thread_start(void*) /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239:28
        #3 0xffffb66c2830 in start_thread (/lib64/libc.so.6+0x90830) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
        #4 0xffffb6666e58 in thread_start (/lib64/libc.so.6+0x34e58) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
    AddressSanitizer can not provide additional info.
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    SUMMARY: AddressSanitizer: SEGV /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238:5 in __sweep_expire
    Thread T10 created by T0 here:
        #0 0xffffb7eecb74 in pthread_create /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:250:3
        #1 0xffffb732ef94 in __wt_thread_create /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/os_posix/os_thread.c:71:5
        #2 0xffffb6f9a37c in __wti_sweep_create /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:664:5
        #3 0xffffb6f7f184 in __wti_connection_workers /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_open.c:316:5
        #4 0xffffb6eb7c24 in wiredtiger_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_api.c:3836:5
        #5 0xaaaabd4c7304 in testutil_wiredtiger_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/utility/misc.c:406:5
        #6 0xaaaabd4aef2c in node_open /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/node.c:122:5
        #7 0xaaaabd4aea98 in node_main /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/node.c:454:5
        #8 0xaaaabd4aaf8c in main /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/main.c:450:17
        #9 0xffffb6667a74 in __libc_start_call_main (/lib64/libc.so.6+0x35a74) (BuildId: f62e89dfc0895638abb4fdaf98b2630da7220235)
        #10 0xaffffb6667b58  (<unknown module>)
        #11 0x2eaaaabd4a19ec  (<unknown module>)
    ==17463==ABORTING
    AddressSanitizer:DEADLYSIGNAL
    AddressSanitizer:DEADLYSIGNAL
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    test_schema_disagg_abort: FAILED: node0 terminated unexpectedly: signal 6: Invalid argument
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    Running task commands failed: running command: command failed: shell script encountered problem: exit code 134
    Finished running task commands in 1h8m0.961166712s.
    Task completed - FAILURE.
    Running post-task commands.
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    #0  0x0000ffff91cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6
    #0  0x0000ffff91cc4454 in __pthread_kill_implementation () from /lib64/libc.so.6
    #1  0x0000ffff91c7b320 [PAC] in raise () from /lib64/libc.so.6
    #2  0x0000ffff91c62224 [PAC] in abort () from /lib64/libc.so.6
    #3  0x0000ffff929033f0 [PAC] in __wt_abort (session=0x0) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/os_common/os_abort.c:32
    #4  0x0000aaaace083edc in testutil_die (e=22, fmt=0xaaaace09b140 <str> "%s terminated unexpectedly: %s %d") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/utility/misc.c:75
    #5  0x0000aaaace0751cc in die_on_child_status (proc=0xaaaace1411c0 <run_children[children]>, status=SUBPROC_KILLED, code=6) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:140
    #6  0x0000aaaace074914 in child_alive (proc=0xaaaace1411c0 <run_children[children]>) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:157
    #7  0x0000aaaace0746ec in children_poll (children=0xaaaace1411c0 <run_children[children]>) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:171
    #8  0x0000aaaace072248 in run_children (cfg=0xffff90200020, self_path=0xffff90202190 "/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:313
    #9  0x0000aaaace071714 in parent_main (cfg=0xffff90200020, self_path=0xffff90202190 "/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/test/csuite/schema_disagg_abort/test_schema_disagg_abort") at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/parent.c:407
    #10 0x0000aaaace06b044 in main (argc=14, argv=0xffffdd9fa008) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/test/csuite/schema_disagg_abort/main.c:461
    #0  0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
    #0  0x0000ffffb66c4454 in __pthread_kill_implementation () from /lib64/libc.so.6
    #1  0x0000ffffb667b320 [PAC] in raise () from /lib64/libc.so.6
    #2  0x0000ffffb6662224 [PAC] in abort () from /lib64/libc.so.6
    #3  0x0000ffffb7e5127c [PAC] in Abort () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp:163
    #4  0x0000ffffb7e4f164 in __sanitizer::Die() () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:58
    #5  0x0000ffffb7f0cac0 in ~ScopedInErrorReport () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:193
    #6  0x0000ffffb7f0c418 in ReportDeadlySignal () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_report.cpp:221
    #7  0x0000ffffb7f0b94c in AsanOnDeadlySignal () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_posix.cpp:40
    #8  <signal handler called>
    #9  0x0000ffffb6f9d758 in __sweep_expire (session=0xffffb5e128c0, now=1789432419) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:238
    #10 0x0000ffffb6f9abfc in __sweep_server (arg=0xffffb5e128c0) at /data/mci/adf3b7f0cee2df31cbeaa897bb216b28/wiredtiger/src/conn/conn_sweep.c:561
    #11 0x0000ffffb7f056e0 in asan_thread_start () at /data/mci/3fc465e6a85d2b4e52ac6fc50781c0fd/toolchain-builder/tmp/build-llvm-v5.sh-aQL/llvm-project-llvmorg/compiler-rt/lib/asan/asan_interceptors.cpp:239
    #12 0x0000ffffb66c2834 in start_thread () from /lib64/libc.so.6
    #13 0x0000ffffb6666e5c [PAC] in thread_start () from /lib64/libc.so.6
    

    logs

    schema-disagg-abort-unique-switch-test-disagg-1 task_log

    Logs:

    Finished command 'shell.exec' in function 'dump stacktraces' (step 1 of 9) in block 'post' in 2.852865817s.
    Running command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post'.
    Putting files matching filter '[wiredtiger/cmake_build/*stacktrace.txt]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
    Finished command 's3.put' in function 'upload stacktraces' (step 2 of 9) in block 'post' in 196.457792ms.
    Running command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post'.
    Putting files matching filter '[wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
    File filter 'wiredtiger/cmake_build/test/format/RUNDIR*/CONFIG' matched no files.
    Finished command 's3.put' in function 'upload test/format config' (step 3 of 9) in block 'post' in 52.09099ms.
    Running command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post'.
    Putting files matching filter '[wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload]' into path 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in S3 bucket 'build_external'.
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/' in bucket 'build_external' (attempt 1 of 5).
    File filter 'wiredtiger/cmake_build/test/model/tools/WT_TEST*/*.workload' matched no files.
    Finished command 's3.put' in function 'upload test/model workloads' (step 4 of 9) in block 'post' in 46.840821ms.
    Running command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post'.
    if [ -d "WT_TEST" ]; then
      # Dump stderr/stdout contents generated by the C libraries onto console for Python tests
      find "WT_TEST" -name "std*.txt" ! -empty -exec bash -c "echo 'Contents from {}:'; cat '{}'" \;
    fi
    Finished command 'shell.exec' in function 'dump stderr/stdout' (step 5 of 9) in block 'post' in 1.383234ms.
    Running command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post'.
    mkdir stat_files
    python3 ../test/evergreen/collect_stat_files.py -d stat_files
    Finished command 'shell.exec' in function 'upload stat files' (step 6.1 of 9) in block 'post' in 57.728629ms.
    Running command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post'.
    Beginning to build archive.
    [message='successfully created archive' num_files='4' target='/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger/cmake_build/stat_files.tar.gz']
    Finished command 'archive.targz_pack' in function 'upload stat files' (step 6.2 of 9) in block 'post' in 29.543771ms.
    Running command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post'.
    Putting local file 'wiredtiger/cmake_build/stat_files.tar.gz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz).
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29/stat_files.tar.gz' in bucket 'build_external' (attempt 1 of 5).
    Finished command 's3.put' in function 'upload stat files' (step 6.3 of 9) in block 'post' in 370.35074ms.
    Running command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post'.
    Beginning to build archive.
    [message='successfully created archive' num_files='8595' target='/data/mci/83c1a8aa392bffbef0a877d0d1152470/wiredtiger.tgz']
    Finished command 'archive.targz_pack' in function 'upload artifact' (step 7.1 of 9) in block 'post' in 1.662733423s.
    Running command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post'.
    Putting local file 'wiredtiger.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz).
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/artifacts/schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29-0.tgz' in bucket 'build_external' (attempt 1 of 5).
    Finished command 's3.put' in function 'upload artifact' (step 7.2 of 9) in block 'post' in 1.434121765s.
    Running command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post'.
    Beginning to build archive.
    No files were archived.
    Finished command 'archive.targz_pack' in function 'save wt hang analyzer core/debugger files' (step 8.1 of 9) in block 'post' in 491.922µs.
    Running command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post'.
    Putting local file 'wt-hang-analyzer.tgz' into path 'build_external/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' (https://build_external.s3.amazonaws.com/wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz).
    Performing S3 put to file 'wiredtiger/amazon2023-disagg-asan-stress/c29c44c5672042f6c2b77155c2c4a2ad342e5c1e/wt_hang_analyzer/wt-hang-analyzer_schema-disagg-abort-unique-switch-test-disagg-1_wiredtiger_disagg_amazon2023_disagg_asan_stress_c29c44c5672042f6c2b77155c2c4a2ad342e5c1e_26_09_14_21_57_29.tgz' in bucket 'build_external' (attempt 1 of 5).
    File '/data/mci/83c1a8aa392bffbef0a877d0d1152470/wt-hang-analyzer.tgz' not found and skip missing is true, exiting without error.
    Finished command 's3.put' in function 'save wt hang analyzer core/debugger files' (step 8.2 of 9) in block 'post' in 14.105694ms.
    Running command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post'.
    Finished command 'shell.exec' in function 'cleanup' (step 9 of 9) in block 'post' in 502.273724ms.
    Finished running post-task commands in 7.222493456s.
    reading test log spec; falling back to default spec: open /data/mci/83c1a8aa392bffbef0a877d0d1152470/build/TestLogs/log_spec.yaml: no such file or directory
    

    logs

          Assignee:
          Chenhao Qu
          Reporter:
          xgen-buildbaron-user
          Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

            Created:
            Updated:
            Resolved: