-
Type:
Build Failure
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Disagg CI-blocker
-
Security Level: Public (Available to anyone on the web)
-
Storage Engines - Foundations
-
528.394
-
SE Foundations - 2026-05-22
-
5
-
8
format-stress-test-disagg-switch-data-validation-2 on amazon2023-disagg-stress
Host: i-0743b2f334fe4a95c
Project: wiredtiger-disagg
Commit: db50b473
Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules
Task Logs:
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
Task logger initialized (agent version '2026-04-15b' from Evergreen build revision '16e97676a2de47b3c6016006969a416f5a3d8d8f').
Starting task 'wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_2_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00', execution 0.
Running pre-task commands.
Running command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre'.
Finished command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre' in 2.518159ms.
Running command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre'.
Finished command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre' in 46.085µs.
Finished running pre-task commands in 2.756242ms.
Running task commands.
Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
Fetching remote file 'wiredtiger/amazon2023-disagg-stress/db50b4733b8c817ad13215a9c0f3028751e5d4c8/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_stress_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 5.20552294s.
Running command 'shell.exec' in function 'format test disagg' (step 2 of 2).
export WT_TOPDIR=$(git rev-parse --show-toplevel)
export WT_BUILDDIR=$WT_TOPDIR/cmake_build
if [ "$OS" = "Windows_NT" ]; then
export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
else
export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
export LD_LIBRARY_PATH=$WT_BUILDDIR
fi
# Create the common sanitizer options and export the specific sanitizer environment
# variables.
COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
if [[ "" =~ ASan ]]; then
export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname $(/opt/mongodbtoolchain/v5/bin/clang -print-file-name=libclang_rt.asan.so))"
export TESTUTIL_BYPASS_ASAN=1
elif [[ "" =~ MSan ]]; then
export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
export TESTUTIL_MSAN=1
elif [[ "" =~ TSan ]]; then
export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
export TESTUTIL_TSAN=1
elif [[ "" =~ UBSan ]]; then
export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
export TESTUTIL_UBSAN=1
fi
if [[ 1 -eq 1 ]]; then
# The preloaded library should be downloaded/built during "configure wiredtiger" step.
# DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
# and as close as possible to the test execution, since it affects all the binaries.
export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
if [ ! -f "$LD_PRELOAD" ]; then
echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
exit 1
fi
fi
# Set extensions to load
EXT="extensions=["
EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
EXT="$EXT ]"
export EXT
Dump Environment
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
BASH_FUNC_which%%=() { ( alias;
CI=true
EVR_AGENT_PID=3631
EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_2_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00
EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
GOCACHE=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build/test/format/.gocache
GOTRACEBACK=none
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/ec2-user
HOSTNAME=ip-10-122-61-18.ec2.internal
INVOCATION_ID=8a299ced9de5402b984fb6a27747085e
JASPER_ID=3fa24df3-59ef-4fbf-afac-de335a2f042a
JASPER_MANAGER=40bfc1d8-72e6-4a7b-812c-fd046425ac02
JOURNAL_STREAM=8:23845
LANG=C.UTF-8
LC_ALL=C
LD_LIBRARY_PATH=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build
LD_PRELOAD=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/TCMALLOC_LIB/libtcmalloc.so
LESSOPEN=||/usr/bin/lesspipe.sh %s
LOGNAME=ec2-user
MAIL=/var/spool/mail/ec2-user
PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
PWD=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build/test/format
SHELL=/bin/bash
SHLVL=1
SYSTEMD_COLORS=false
SYSTEMD_EXEC_PID=3540
S_COLORS=auto
TEMP=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
TMP=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
TMPDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
USER=ec2-user
WT_BUILDDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build
WT_TOPDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger
_=/usr/bin/printenv
which_declare=declare -f
}
echo "Dump Environment"
printenv | sort
# Fail, show the configuration file.
fail() {
echo "======= FAILURE =========="
[ -f RUNDIR/CONFIG ] && cat RUNDIR/CONFIG
exit 1
}
for i in $(seq 5); do
echo Iteration $i/5
rm -rf RUNDIR
./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch ops.verify=1 runs.mirror=1 table1.runs.source=table table1.disagg.enabled=0 runs.timer=5:10 || fail
Iteration 1/5
t: process 236477 running
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=587716,metadata_checksum=4e3317aa,database_size=696548523,version=1,compatible_version=1],timestamp=0x5c43be) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1113622,metadata_checksum=6a147d98,database_size=757174730,version=1,compatible_version=1],timestamp=0x7c2a22) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1490154,metadata_checksum=adcb559b,database_size=744381894,version=1,compatible_version=1],timestamp=0x97a16d) ---
t: successful run completed (477 seconds)
if [ true = true ]; then
./t -R $format_args || fail
fi
done
t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls
t: WARNING: table.1 skipped verify because of EBUSY
t: WARNING: table.2 skipped verify because of EBUSY
t: process 236912 running
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1781928,metadata_checksum=1a5db1e6,database_size=828605300,version=1,compatible_version=1],timestamp=0xc063fe) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2074623,metadata_checksum=7a56cb06,database_size=847816127,version=1,compatible_version=1],timestamp=0xeeba68) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2346437,metadata_checksum=9603638d,database_size=801846089,version=1,compatible_version=1],timestamp=0x11f84fd) ---
t: successful run completed (515 seconds)
t: WARNING: table.1 skipped verify because of EBUSY
t: WARNING: table.2 skipped verify because of EBUSY
Iteration 2/5
t: process 237325 running
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1296242,metadata_checksum=4f840ddd,database_size=360999059,version=1,compatible_version=1],timestamp=0x4b9c0d) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2106100,metadata_checksum=af31aca6,database_size=354001560,version=1,compatible_version=1],timestamp=0x6c7ab2) ---
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2863707,metadata_checksum=6a40f9e6,database_size=346117599,version=1,compatible_version=1],timestamp=0x8ab99d) ---
t: successful run completed (486 seconds)
t: WARNING: table.3 skipped verify because of EBUSY
t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls
t: process 237785 running
0x737abfca0378:[1776827681:121404][237785:0xffff24d6dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2441770 pinned in session 20 [last_op: WT_CURSOR.prev] (current_id 2452680)
0x737abfc9dc40:[1776827690:083973][237785:0xffff2557dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2625672 pinned in session 20 [last_op: WT_CURSOR.search] (current_id 2639598)
0x737abfca0378:[1776827718:297757][237785:0xffff24d6dbc0], t, file:T00003.wt_stable, WT_CURSOR.update: [WT_VERB_TRANSACTION][NOTICE]: oldest id 3992655 pinned in session 20 [last_op: WT_CURSOR.get_value] (current_id 4003032)
0x737abfca0b50:[1776827719:795248][237785:0xffff2455dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 4141319 pinned in session 20 [last_op: WT_CURSOR.remove] (current_id 4152086)
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=3493959,metadata_checksum=8f62927e,database_size=346435180,version=1,compatible_version=1],timestamp=0xc78658) ---
0x737abfc9cc90:[1776827751:083313][237785:0xffff653ddbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 203707 pinned in session 20 [last_op: WT_CURSOR.prev] (current_id 236336)
0x737abfc9d468:[1776827785:646149][237785:0xffff6135dbc0], t, file:T00003.wt_stable, WT_CURSOR.insert: [WT_VERB_TRANSACTION][NOTICE]: oldest id 1729074 pinned in session 25 [last_op: WT_CURSOR.reset] (current_id 1739324)
0x737abfca1b00:[1776827805:143988][237785:0xffff6339dbc0], t, file:T00003.wt_stable, WT_CURSOR.update: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2701945 pinned in session 24 [last_op: WT_CURSOR.update] (current_id 2711962)
--- [Follower] Picked up checkpoint (metadata=[metadata_lsn=3916081,metadata_checksum=bbfd05f1,database_size=332312474,version=1,compatible_version=1],timestamp=0xfba6a8) ---
0x737af129b468:[1776827867:136294][237785:0xffff64bddbc0], t, file:T00003.wt_ingest, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 14048 pinned in session 25 [last_op: WT_CURSOR.next] (current_id 27029)
0x737af129dba0:[1776827868:488
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
[1776829991:487719][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
0x32edb589ff60:transaction state dump 0x32edb589ff60:current ID: 748318 0x32edb589ff60:last running ID: 748318 0x32edb589ff60:metadata_pinned ID: 748318 0x32edb589ff60:oldest ID: 748318 0x32edb589ff60:durable timestamp: (0, 19296225) 0x32edb589ff60:oldest timestamp: (0, 19043709) 0x32edb589ff60:pinned timestamp: (0, 19043709) 0x32edb589ff60:stable timestamp: (0, 19043709) 0x32edb589ff60:has_durable_timestamp: yes 0x32edb589ff60:has_oldest_timestamp: yes 0x32edb589ff60:has_pinned_timestamp: yes 0x32edb589ff60:has_stable_timestamp: yes 0x32edb589ff60:oldest_is_pinned: yes 0x32edb589ff60:stable_is_pinned: yes 0x32edb589ff60:checkpoint running: no 0x32edb589ff60:checkpoint generation: 1 0x32edb589ff60:checkpoint pinned ID: 0 0x32edb589ff60:checkpoint txn ID: 0 0x32edb589ff60:session count: 17 0x32edb589ff60:Transaction state of active sessions: 0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x32edb589ff60:cache dump 0x32edb589ff60:cache full: no 0x32edb589ff60:cache clean check: no (24.909%) 0x32edb589ff60:cache dirty check: yes (24.432%) 0x32edb589ff60:cache updates check: no (6.732%) 0x32edb589ff60:file:T00002.wt_stable(<live>): 0x32edb589ff60:internal: 33 pages, 2730.14 KB, 4/29 clean/dirty pages, 1527.57/1202.58 clean / dirty KB, 642.27 KB max page, 116.06 KB max dirty page 0x32edb589ff60:leaf: 3085 pages, 360611.61 KB, 59/3026 clean/dirty pages, 8137.69 /352473.92 /85052.50 clean/dirty/updates KB, 562.76 KB max page, 562.76 KB max dirty page 0x32edb589ff60:file:T00003.wt_stable(<live>): 0x32edb589ff60:internal: 38 pages, 1362.41 KB, 11/27 clean/dirty pages, 1222.73/139.68 clean / dirty KB, 401.68 KB max page, 81.79 KB max dirty page 0x32edb589ff60:leaf: 172 pages, 5379.70 KB, 2/170 clean/dirty pages, 14.49 /5365.21 /4244.37 clean/dirty/updates KB, 256.04 KB max page, 256.04 KB max dirty page 0x32edb589ff60:file:T00003.wt_ingest(<live>): 0x32edb589ff60:internal: 1 pages, 0.74 KB, 0/1 clean/dirty pages, 0.00/0.74 clean / dirty KB, 0.74 KB max page, 0.74 KB max dirty page 0x32edb589ff60:leaf: 0 pages 0x32edb589ff60:file:T00002.wt_ingest(<live>): 0x32edb589ff60:internal: 1 pages, 0.74 KB, 0/1 clean/dirty pages, 0.00/0.74 clean / dirty KB, 0.74 KB max page, 0.74 KB max dirty page 0x32edb589ff60:leaf: 1 pages, 0.72 KB, 0/1 clean/dirty pages, 0.00 /0.72 /0.39 clean/dirty/updates KB, 0.72 KB max page, 0.72 KB max dirty page 0x32edb589ff60:file:T00001.wt(<live>): 0x32edb589ff60:internal: 55 pages, 1089.94 KB, 5/50 clean/dirty pages, 331.50/758.44 clean / dirty KB, 161.01 KB max page, 91.01 KB max dirty page 0x32edb589ff60:leaf: 2673 pages, 353651.14 KB, 0/2673 clean/dirty pages, 0.00 /353651.14 /106739.89 clean/dirty/updates KB, 822.41 KB max page, 822.41 KB max dirty page 0x32edb589ff60:file:WiredTigerSharedHS.wt_stable(<live>): 0x32edb589ff60:internal: 1 pages, 0.94 KB, 1/0 clean/dirty pages, 0.94/0.00 clean / dirty KB, 0.94 KB max page, 0.00 KB max dirty page 0x32edb589ff60:leaf: 3 pages, 519.21 KB, 3/0 clean/dirty pages, 519.21 /0.00 /0.00 clean/dirty/updates KB, 177.19 KB max page, 0.00 KB max dirty page 0x32edb589ff60:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.40(<live>): 0x32edb589ff60:internal: 1 pages, 0.94 KB, 1/0 clean/dirty pages, 0.94/0.00 clean / dirty KB, 0.94 KB max page, 0.00 KB max dirty page 0x32edb589ff60:leaf: 0 pages 0x32edb589ff60:file:WiredTigerHS.wt(<live>): 0x32edb589ff60:internal: 2 pages, 2.84 KB, 1/1 clean/dirty pages, 2.06/0.78 clean / dirty KB, 2.06 KB max page, 0.78 KB max dirty page 0x32edb589ff60:leaf: 9 pages, 7.20 KB, 9/0 clean/dirty pages, 7.20 /0.00 /0.00 clean/dirty/updates KB, 1.89 KB max page, 0.00 KB max dirty page 0x32edb589ff60:file:WiredTiger.wt(<live>): 0x32edb589ff60:internal: 1 pages, 0.77 KB, 0/1 clean/dirty pages, 0.00/0.77 clean / dirty KB, 0.77 KB max page, 0.77 KB max dirty page 0x32edb589ff60:leaf: 1 pages, 13.97 KB, 1/0 clean/dirty pages, 13.97 /0.00 /0.00 clean/dirty/updates KB, 13.97 KB max page, 0.00 KB max dirty page 0x32edb589ff60:cache dump: total found: 765.04 MB vs tracked inuse 708.51 MB 0x32edb589ff60:total dirty bytes: 696.87 MB vs tracked dirty 697.01 MB
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
0x32edb589ff60:total updates bytes: 191.44 MB vs tracked updates 206.79 MB 0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= 0x32edb589ff60:metadata dump 0x32edb589ff60:uri: colgroup:T00001 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="file:T00001.wt",type=file,verbose=[],write_timestamp_usage=none 0x32edb589ff60:uri: colgroup:T00002 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="layered:T00002",type=layered,verbose=[],write_timestamp_usage=none 0x32edb589ff60:uri: colgroup:T00003 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="layered:T00003",type=layered,verbose=[],write_timestamp_usage=none 0x32edb589ff60:uri: file:T00001.wt | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checksum="off",collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=16,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=2048,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=2906,leaf_page_max=131072,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=7340032,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=3,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=67,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.43=(addr="01e3121b17c101e41eb529a5c87d93e47e56853ad1cd99e446355d8f808080e4247911c0e41612efc0",order=43,time=1776829420,size=370512384,newest_start_durable_ts=18698518,oldest_start_ts=0,newest_txn=8850100,newest_stop_durable_ts=0,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1460263,run_write_gen=1084663,next_page_id=100)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647) 0x32edb589ff60:uri: file:T00002.wt_ingest | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,checksum="unencrypted",collator=,columns=,dictionary=0,disaggregated=(page_log=none,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=24,ignore_in_memory_cache_size=false,in_memory=true,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=131072,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=3015,leaf_page_max=131072,leaf_value_max=0,live_restore=(bitmap=,nbits=0),log=(enabled=false),memory_page_image_max=0,memory_page_max=2097152,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=5,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=70,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none 0x32edb589ff60:uri: file:T00002.wt_stable | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=disagg,cache_resident=false,checksum="unencrypted",collator=,columns=,dictionary=0,disaggregated=(page_log=palite,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=33,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=131072,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=3015,leaf_page_max=131072,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=2097152,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=5,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=70,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.42=(addr="00dc1e80e312d3cfe312d3cfe3026061e706fb68",order=42,time=1776829420,size=450993881,newest_start_durable_ts=18698263,oldest_start_ts=0,newest_txn=8849752,newest_stop_durable_ts=17229971,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1260536,run_write_gen=1084663,next_page_id=7263)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647) 0x32edb589ff60:uri: file:T00003.wt_ingest | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,checksum="unencrypted",collator=,columns=,dictionary=132,disaggregated=(page_log=none,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=40,ignore_in_memory_cache_size=false,in_memory=true,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=512,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=354,leaf_page_max=16384,leaf_value_max=0,live_restore=(bitmap=,nbits=0),log=(enabled=false),memory_page_image_max=0,memory_page_max=1048576,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=6,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=96,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none 0x32edb589ff60:uri: file:T00003.wt_stable | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=disagg,cache_resident=false,checksum="unencrypted",collator=,columns=,dictionary=132,disaggregated=(page_log=palite,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=49,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=512,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=354,leaf_page_max=16384,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=1048576,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=6,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=96,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.43=(addr="00e2cd8e80e312d3dde312d3dde30620ddd252399a",order=43,time=1776829420,size=272098222,newest_start_durable_ts=18697200,oldest_start_ts=0,newest_txn=8848407,newest_stop_durable_ts=17169611,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1092667,run_write_gen=1084663,next_page_id=60879)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647) 0x32edb589ff60:uri: file:WiredTigerHS.wt | config: access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=none,block_manager=default,cache_resident=false,checksum=on,collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=8,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=16KB,key_format=IuQQ,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),memory_page_image_max=0,memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=QQQu,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.40=(addr="018281e4b96d3d108381e4279ec8a98481e47e1989d1808080e3053fc0e3029fc0",order=40,time=1776829420,size=184320,newest_start_durable_ts=0,oldest_start_ts=0,newest_txn=8756102,newest_stop_durable_ts=18626292,newest_stop_ts=18626292,newest_stop_txn=8756102,prepare=0,write_gen=1085079,run_write_gen=1084663,next_page_id=100)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647) 0x32edb589ff60:uri: file:WiredTigerShared.wt_stable | config: access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_manager=default,cache_resident=false,checksum=on,collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(k
format-stress-test-disagg-switch-data-validation-2 task_log
Logs:
[1776829991:490435][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 795: Error at src/btree/bt_curnext.c:795: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found [1776829991:490444][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found [1776829991:490447][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out [1776829991:490452][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out [1776829991:490455][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic [1776829991:490458][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
- is related to
-
WT-16295 format cache stuck, bt_curnext not found
-
- Open
-