test/format (disagg.mode=switch) cache stuck

XMLWordPrintableJSON

      format-stress-test-disagg-switch-data-validation-2 on amazon2023-disagg-stress

      Host: i-0743b2f334fe4a95c
      Project: wiredtiger-disagg
      Commit: db50b473
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      Task logger initialized (agent version '2026-04-15b' from Evergreen build revision '16e97676a2de47b3c6016006969a416f5a3d8d8f').
      Starting task 'wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_2_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00', execution 0.
      Running pre-task commands.
      Running command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre'.
      Finished command 'shell.exec' in function 'cleanup' (step 1 of 2) in block 'pre' in 2.518159ms.
      Running command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre'.
      Finished command 'expansions.update' in function 'setup environment' (step 2 of 2) in block 'pre' in 46.085µs.
      Finished running pre-task commands in 2.756242ms.
      Running task commands.
      Running command 's3.get' in function 'fetch artifacts' (step 1 of 2).
      Fetching remote file 'wiredtiger/amazon2023-disagg-stress/db50b4733b8c817ad13215a9c0f3028751e5d4c8/artifacts/compile_wiredtiger_disagg_amazon2023_disagg_stress_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00.tgz' from S3 bucket 'build_external' (attempt 1 of 5).
      Finished command 's3.get' in function 'fetch artifacts' (step 1 of 2) in 5.20552294s.
      Running command 'shell.exec' in function 'format test disagg' (step 2 of 2).
      export WT_TOPDIR=$(git rev-parse --show-toplevel)
      export WT_BUILDDIR=$WT_TOPDIR/cmake_build
      if [ "$OS" = "Windows_NT" ]; then
        export PATH=/cygdrive/c/python/Python311:/cygdrive/c/python/Python311/Scripts:$PATH
      else
        export PATH=/opt/mongodbtoolchain/v5/bin:$PATH
        export LD_LIBRARY_PATH=$WT_BUILDDIR
      fi
      # Create the common sanitizer options and export the specific sanitizer environment
      # variables.
      COMMON_SAN_OPTIONS="abort_on_error=1:disable_coredump=0"
      if [[ "" =~ ASan ]]; then
        export ASAN_OPTIONS="$COMMON_SAN_OPTIONS:unmap_shadow_on_exit=1"
        export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(dirname $(/opt/mongodbtoolchain/v5/bin/clang -print-file-name=libclang_rt.asan.so))"
        export TESTUTIL_BYPASS_ASAN=1
      elif [[ "" =~ MSan ]]; then
        export MSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_MSAN=1
      elif [[ "" =~ TSan ]]; then
        export TSAN_OPTIONS="$COMMON_SAN_OPTIONS:verbosity=3"
        export TESTUTIL_TSAN=1
      elif [[ "" =~ UBSan ]]; then
        export UBSAN_OPTIONS="$COMMON_SAN_OPTIONS:print_stacktrace=1"
        export TESTUTIL_UBSAN=1
      fi
      if [[ 1 -eq 1 ]]; then
        # The preloaded library should be downloaded/built during "configure wiredtiger" step.
        # DO NOT MOVE:This variable must be set in the same function(shell) that runs a test,
        # and as close as possible to the test execution, since it affects all the binaries.
        export LD_PRELOAD=$WT_TOPDIR/TCMALLOC_LIB/libtcmalloc.so
        if [ ! -f "$LD_PRELOAD" ]; then
          echo "Error: TCMalloc support was requested, but the library doesn't exist: $LD_PRELOAD"
          exit 1
        fi
      fi
      # Set extensions to load
      EXT="extensions=["
      EXT="$EXT ext/compressors/snappy/libwiredtiger_snappy.so,"
      EXT="$EXT ext/collators/reverse/libwiredtiger_reverse_collator.so,"
      EXT="$EXT ext/encryptors/rotn/libwiredtiger_rotn.so,"
      EXT="$EXT ext/page_log/palite/libwiredtiger_palite.so,"
      EXT="$EXT ]"
      export EXT
      Dump Environment
       eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot "$@"
      BASH_FUNC_which%%=() {  ( alias;
      CI=true
      EVR_AGENT_PID=3631
      EVR_TASK_ID=wiredtiger_disagg_amazon2023_disagg_stress_format_stress_test_disagg_switch_data_validation_2_db50b4733b8c817ad13215a9c0f3028751e5d4c8_26_04_22_01_25_00
      EXT=extensions=[ ext/compressors/snappy/libwiredtiger_snappy.so, ext/collators/reverse/libwiredtiger_reverse_collator.so, ext/encryptors/rotn/libwiredtiger_rotn.so, ext/page_log/palite/libwiredtiger_palite.so, ]
      GOCACHE=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build/test/format/.gocache
      GOTRACEBACK=none
      HISTCONTROL=ignoredups
      HISTSIZE=1000
      HOME=/home/ec2-user
      HOSTNAME=ip-10-122-61-18.ec2.internal
      INVOCATION_ID=8a299ced9de5402b984fb6a27747085e
      JASPER_ID=3fa24df3-59ef-4fbf-afac-de335a2f042a
      JASPER_MANAGER=40bfc1d8-72e6-4a7b-812c-fd046425ac02
      JOURNAL_STREAM=8:23845
      LANG=C.UTF-8
      LC_ALL=C
      LD_LIBRARY_PATH=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build
      LD_PRELOAD=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/TCMALLOC_LIB/libtcmalloc.so
      LESSOPEN=||/usr/bin/lesspipe.sh %s
      LOGNAME=ec2-user
      MAIL=/var/spool/mail/ec2-user
      PATH=/opt/mongodbtoolchain/v5/bin:/usr/sbin:/usr/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/node/bin:/opt/node/bin:/opt/node/bin
      PWD=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build/test/format
      SHELL=/bin/bash
      SHLVL=1
      SYSTEMD_COLORS=false
      SYSTEMD_EXEC_PID=3540
      S_COLORS=auto
      TEMP=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
      TMP=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
      TMPDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/tmp
      USER=ec2-user
      WT_BUILDDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger/cmake_build
      WT_TOPDIR=/data/mci/2e3fb8e8ba44c0ee1fd3cdfbc67235a6/wiredtiger
      _=/usr/bin/printenv
      which_declare=declare -f
      }
      echo "Dump Environment"
      printenv | sort
      # Fail, show the configuration file.
      fail() {
        echo "======= FAILURE =========="
        [ -f RUNDIR/CONFIG ] && cat RUNDIR/CONFIG
        exit 1
      }
      for i in $(seq 5); do
        echo Iteration $i/5
        rm -rf RUNDIR
        ./t -c ../../../test/format/CONFIG.disagg disagg.mode=switch ops.verify=1 runs.mirror=1 table1.runs.source=table table1.disagg.enabled=0 runs.timer=5:10  || fail
      Iteration 1/5
      t: process 236477 running
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=587716,metadata_checksum=4e3317aa,database_size=696548523,version=1,compatible_version=1],timestamp=0x5c43be) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1113622,metadata_checksum=6a147d98,database_size=757174730,version=1,compatible_version=1],timestamp=0x7c2a22) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1490154,metadata_checksum=adcb559b,database_size=744381894,version=1,compatible_version=1],timestamp=0x97a16d) ---
      t: successful run completed (477 seconds)
        if [ true = true ]; then
          ./t -R $format_args  || fail
        fi
      done
      t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls
      t: WARNING: table.1 skipped verify because of EBUSY
      t: WARNING: table.2 skipped verify because of EBUSY
       t: process 236912 running
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1781928,metadata_checksum=1a5db1e6,database_size=828605300,version=1,compatible_version=1],timestamp=0xc063fe) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2074623,metadata_checksum=7a56cb06,database_size=847816127,version=1,compatible_version=1],timestamp=0xeeba68) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2346437,metadata_checksum=9603638d,database_size=801846089,version=1,compatible_version=1],timestamp=0x11f84fd) ---
      t: successful run completed (515 seconds)
      t: WARNING: table.1 skipped verify because of EBUSY
      t: WARNING: table.2 skipped verify because of EBUSY
       Iteration 2/5
      t: process 237325 running
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=1296242,metadata_checksum=4f840ddd,database_size=360999059,version=1,compatible_version=1],timestamp=0x4b9c0d) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2106100,metadata_checksum=af31aca6,database_size=354001560,version=1,compatible_version=1],timestamp=0x6c7ab2) ---
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=2863707,metadata_checksum=6a40f9e6,database_size=346117599,version=1,compatible_version=1],timestamp=0x8ab99d) ---
      t: successful run completed (486 seconds)
      t: WARNING: table.3 skipped verify because of EBUSY
      t: WARNING: turning off ops.random_cursor with disagg as they are currently problematic and can cause stalls
       t: process 237785 running
      0x737abfca0378:[1776827681:121404][237785:0xffff24d6dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2441770 pinned in session 20 [last_op: WT_CURSOR.prev] (current_id 2452680)
      0x737abfc9dc40:[1776827690:083973][237785:0xffff2557dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2625672 pinned in session 20 [last_op: WT_CURSOR.search] (current_id 2639598)
      0x737abfca0378:[1776827718:297757][237785:0xffff24d6dbc0], t, file:T00003.wt_stable, WT_CURSOR.update: [WT_VERB_TRANSACTION][NOTICE]: oldest id 3992655 pinned in session 20 [last_op: WT_CURSOR.get_value] (current_id 4003032)
      0x737abfca0b50:[1776827719:795248][237785:0xffff2455dbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 4141319 pinned in session 20 [last_op: WT_CURSOR.remove] (current_id 4152086)
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=3493959,metadata_checksum=8f62927e,database_size=346435180,version=1,compatible_version=1],timestamp=0xc78658) ---
      0x737abfc9cc90:[1776827751:083313][237785:0xffff653ddbc0], t, file:T00001.wt, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 203707 pinned in session 20 [last_op: WT_CURSOR.prev] (current_id 236336)
      0x737abfc9d468:[1776827785:646149][237785:0xffff6135dbc0], t, file:T00003.wt_stable, WT_CURSOR.insert: [WT_VERB_TRANSACTION][NOTICE]: oldest id 1729074 pinned in session 25 [last_op: WT_CURSOR.reset] (current_id 1739324)
      0x737abfca1b00:[1776827805:143988][237785:0xffff6339dbc0], t, file:T00003.wt_stable, WT_CURSOR.update: [WT_VERB_TRANSACTION][NOTICE]: oldest id 2701945 pinned in session 24 [last_op: WT_CURSOR.update] (current_id 2711962)
      --- [Follower] Picked up checkpoint (metadata=[metadata_lsn=3916081,metadata_checksum=bbfd05f1,database_size=332312474,version=1,compatible_version=1],timestamp=0xfba6a8) ---
      0x737af129b468:[1776827867:136294][237785:0xffff64bddbc0], t, file:T00003.wt_ingest, WT_CURSOR.reset: [WT_VERB_TRANSACTION][NOTICE]: oldest id 14048 pinned in session 25 [last_op: WT_CURSOR.next] (current_id 27029)
      0x737af129dba0:[1776827868:488
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      [1776829991:487719][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 543: Cache stuck for too long, giving up: Connection timed out
      
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      0x32edb589ff60:transaction state dump
      0x32edb589ff60:current ID: 748318
      0x32edb589ff60:last running ID: 748318
      0x32edb589ff60:metadata_pinned ID: 748318
      0x32edb589ff60:oldest ID: 748318
      0x32edb589ff60:durable timestamp: (0, 19296225)
      0x32edb589ff60:oldest timestamp: (0, 19043709)
      0x32edb589ff60:pinned timestamp: (0, 19043709)
      0x32edb589ff60:stable timestamp: (0, 19043709)
      0x32edb589ff60:has_durable_timestamp: yes
      0x32edb589ff60:has_oldest_timestamp: yes
      0x32edb589ff60:has_pinned_timestamp: yes
      0x32edb589ff60:has_stable_timestamp: yes
      0x32edb589ff60:oldest_is_pinned: yes
      0x32edb589ff60:stable_is_pinned: yes
      0x32edb589ff60:checkpoint running: no
      0x32edb589ff60:checkpoint generation: 1
      0x32edb589ff60:checkpoint pinned ID: 0
      0x32edb589ff60:checkpoint txn ID: 0
      0x32edb589ff60:session count: 17
      0x32edb589ff60:Transaction state of active sessions:
      0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x32edb589ff60:cache dump
      0x32edb589ff60:cache full: no
      0x32edb589ff60:cache clean check: no (24.909%)
      0x32edb589ff60:cache dirty check: yes (24.432%)
      0x32edb589ff60:cache updates check: no (6.732%)
      0x32edb589ff60:file:T00002.wt_stable(<live>):
      0x32edb589ff60:internal: 33 pages, 2730.14 KB, 4/29 clean/dirty pages, 1527.57/1202.58 clean / dirty KB, 642.27 KB max page, 116.06 KB max dirty page
      0x32edb589ff60:leaf: 3085 pages, 360611.61 KB, 59/3026 clean/dirty pages, 8137.69 /352473.92 /85052.50 clean/dirty/updates KB, 562.76 KB max page, 562.76 KB max dirty page
      0x32edb589ff60:file:T00003.wt_stable(<live>):
      0x32edb589ff60:internal: 38 pages, 1362.41 KB, 11/27 clean/dirty pages, 1222.73/139.68 clean / dirty KB, 401.68 KB max page, 81.79 KB max dirty page
      0x32edb589ff60:leaf: 172 pages, 5379.70 KB, 2/170 clean/dirty pages, 14.49 /5365.21 /4244.37 clean/dirty/updates KB, 256.04 KB max page, 256.04 KB max dirty page
      0x32edb589ff60:file:T00003.wt_ingest(<live>):
      0x32edb589ff60:internal: 1 pages, 0.74 KB, 0/1 clean/dirty pages, 0.00/0.74 clean / dirty KB, 0.74 KB max page, 0.74 KB max dirty page
      0x32edb589ff60:leaf: 0 pages
      0x32edb589ff60:file:T00002.wt_ingest(<live>):
      0x32edb589ff60:internal: 1 pages, 0.74 KB, 0/1 clean/dirty pages, 0.00/0.74 clean / dirty KB, 0.74 KB max page, 0.74 KB max dirty page
      0x32edb589ff60:leaf: 1 pages, 0.72 KB, 0/1 clean/dirty pages, 0.00 /0.72 /0.39 clean/dirty/updates KB, 0.72 KB max page, 0.72 KB max dirty page
      0x32edb589ff60:file:T00001.wt(<live>):
      0x32edb589ff60:internal: 55 pages, 1089.94 KB, 5/50 clean/dirty pages, 331.50/758.44 clean / dirty KB, 161.01 KB max page, 91.01 KB max dirty page
      0x32edb589ff60:leaf: 2673 pages, 353651.14 KB, 0/2673 clean/dirty pages, 0.00 /353651.14 /106739.89 clean/dirty/updates KB, 822.41 KB max page, 822.41 KB max dirty page
      0x32edb589ff60:file:WiredTigerSharedHS.wt_stable(<live>):
      0x32edb589ff60:internal: 1 pages, 0.94 KB, 1/0 clean/dirty pages, 0.94/0.00 clean / dirty KB, 0.94 KB max page, 0.00 KB max dirty page
      0x32edb589ff60:leaf: 3 pages, 519.21 KB, 3/0 clean/dirty pages, 519.21 /0.00 /0.00 clean/dirty/updates KB, 177.19 KB max page, 0.00 KB max dirty page
      0x32edb589ff60:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.40(<live>):
      0x32edb589ff60:internal: 1 pages, 0.94 KB, 1/0 clean/dirty pages, 0.94/0.00 clean / dirty KB, 0.94 KB max page, 0.00 KB max dirty page
      0x32edb589ff60:leaf: 0 pages
      0x32edb589ff60:file:WiredTigerHS.wt(<live>):
      0x32edb589ff60:internal: 2 pages, 2.84 KB, 1/1 clean/dirty pages, 2.06/0.78 clean / dirty KB, 2.06 KB max page, 0.78 KB max dirty page
      0x32edb589ff60:leaf: 9 pages, 7.20 KB, 9/0 clean/dirty pages, 7.20 /0.00 /0.00 clean/dirty/updates KB, 1.89 KB max page, 0.00 KB max dirty page
      0x32edb589ff60:file:WiredTiger.wt(<live>):
      0x32edb589ff60:internal: 1 pages, 0.77 KB, 0/1 clean/dirty pages, 0.00/0.77 clean / dirty KB, 0.77 KB max page, 0.77 KB max dirty page
      0x32edb589ff60:leaf: 1 pages, 13.97 KB, 1/0 clean/dirty pages, 13.97 /0.00 /0.00 clean/dirty/updates KB, 13.97 KB max page, 0.00 KB max dirty page
      0x32edb589ff60:cache dump: total found: 765.04 MB vs tracked inuse 708.51 MB
      0x32edb589ff60:total dirty bytes: 696.87 MB vs tracked dirty 697.01 MB
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      0x32edb589ff60:total updates bytes: 191.44 MB vs tracked updates 206.79 MB
      0x32edb589ff60:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x32edb589ff60:metadata dump
      0x32edb589ff60:uri: colgroup:T00001 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="file:T00001.wt",type=file,verbose=[],write_timestamp_usage=none
      0x32edb589ff60:uri: colgroup:T00002 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="layered:T00002",type=layered,verbose=[],write_timestamp_usage=none
      0x32edb589ff60:uri: colgroup:T00003 | config: app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),collator=,columns=,source="layered:T00003",type=layered,verbose=[],write_timestamp_usage=none
      0x32edb589ff60:uri: file:T00001.wt | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checksum="off",collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=16,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=2048,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=2906,leaf_page_max=131072,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=7340032,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=3,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=67,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.43=(addr="01e3121b17c101e41eb529a5c87d93e47e56853ad1cd99e446355d8f808080e4247911c0e41612efc0",order=43,time=1776829420,size=370512384,newest_start_durable_ts=18698518,oldest_start_ts=0,newest_txn=8850100,newest_stop_durable_ts=0,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1460263,run_write_gen=1084663,next_page_id=100)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647)
      0x32edb589ff60:uri: file:T00002.wt_ingest | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,checksum="unencrypted",collator=,columns=,dictionary=0,disaggregated=(page_log=none,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=24,ignore_in_memory_cache_size=false,in_memory=true,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=131072,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=3015,leaf_page_max=131072,leaf_value_max=0,live_restore=(bitmap=,nbits=0),log=(enabled=false),memory_page_image_max=0,memory_page_max=2097152,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=5,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=70,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none
      0x32edb589ff60:uri: file:T00002.wt_stable | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=disagg,cache_resident=false,checksum="unencrypted",collator=,columns=,dictionary=0,disaggregated=(page_log=palite,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=33,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=131072,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=3015,leaf_page_max=131072,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=2097152,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=5,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=70,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.42=(addr="00dc1e80e312d3cfe312d3cfe3026061e706fb68",order=42,time=1776829420,size=450993881,newest_start_durable_ts=18698263,oldest_start_ts=0,newest_txn=8849752,newest_stop_durable_ts=17229971,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1260536,run_write_gen=1084663,next_page_id=7263)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647)
      0x32edb589ff60:uri: file:T00003.wt_ingest | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=default,cache_resident=false,checkpoint=,checkpoint_backup_info=,checkpoint_lsn=,checksum="unencrypted",collator=,columns=,dictionary=132,disaggregated=(page_log=none,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=40,ignore_in_memory_cache_size=false,in_memory=true,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=512,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=354,leaf_page_max=16384,leaf_value_max=0,live_restore=(bitmap=,nbits=0),log=(enabled=false),memory_page_image_max=0,memory_page_max=1048576,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=6,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=96,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none
      0x32edb589ff60:uri: file:T00003.wt_stable | config: access_pattern_hint=none,allocation_size=512,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor="none",block_manager=disagg,cache_resident=false,checksum="unencrypted",collator=,columns=,dictionary=132,disaggregated=(page_log=palite,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=49,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=512,key_format=u,key_gap=10,leaf_item_max=0,leaf_key_max=354,leaf_page_max=16384,leaf_value_max=0,log=(enabled=false),memory_page_image_max=0,memory_page_max=1048576,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=true,prefix_compression_min=6,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=96,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=u,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.43=(addr="00e2cd8e80e312d3dde312d3dde30620ddd252399a",order=43,time=1776829420,size=272098222,newest_start_durable_ts=18697200,oldest_start_ts=0,newest_txn=8848407,newest_stop_durable_ts=17169611,newest_stop_ts=-1,newest_stop_txn=-11,prepare=0,write_gen=1092667,run_write_gen=1084663,next_page_id=60879)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647)
      0x32edb589ff60:uri: file:WiredTigerHS.wt | config: access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=none,block_manager=default,cache_resident=false,checksum=on,collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(keyid=,name=),format=btree,huffman_key=,huffman_value=,id=8,ignore_in_memory_cache_size=false,in_memory=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=16KB,key_format=IuQQ,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),memory_page_image_max=0,memory_page_max=5MB,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,readonly=false,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,tiered_object=false,tiered_storage=(auth_token=,bucket=,bucket_prefix=,cache_directory=,local_retention=300,name=,object_target_size=0),value_format=QQQu,verbose=[],version=(major=2,minor=1),write_timestamp_usage=none,checkpoint=(WiredTigerCheckpoint.40=(addr="018281e4b96d3d108381e4279ec8a98481e47e1989d1808080e3053fc0e3029fc0",order=40,time=1776829420,size=184320,newest_start_durable_ts=0,oldest_start_ts=0,newest_txn=8756102,newest_stop_durable_ts=18626292,newest_stop_ts=18626292,newest_stop_txn=8756102,prepare=0,write_gen=1085079,run_write_gen=1084663,next_page_id=100)),checkpoint_backup_info=,checkpoint_lsn=(4294967295,2147483647)
      0x32edb589ff60:uri: file:WiredTigerShared.wt_stable | config: access_pattern_hint=none,allocation_size=4KB,app_metadata=,assert=(commit_timestamp=none,durable_timestamp=none,read_timestamp=none,write_timestamp=off),block_allocation=best,block_compressor=,block_manager=default,cache_resident=false,checksum=on,collator=,columns=,dictionary=0,disaggregated=(page_log=,storage_tier=none),encryption=(k
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      [1776829991:490435][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __wt_btcur_next, 795: Error at src/btree/bt_curnext.c:795: "WT_NOTFOUND" failed: WT_NOTFOUND: item not found
      [1776829991:490444][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __curfile_next, 186: Error at src/cursor/cur_file.c:186: "ret" failed: WT_NOTFOUND: item not found
      [1776829991:490447][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_ERROR_RETURNS][ERROR]: __evict_thread_run, 336: Error at src/evict/evict_lru.c:336: "ret" failed: Connection timed out
      [1776829991:490452][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: eviction thread error: Connection timed out
      [1776829991:490455][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_thread_run, 359: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1776829991:490458][239488:0xffff99bedbc0], t, file:WiredTigerSharedHS.wt_stable, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 29: aborting WiredTiger library
      

      logs

            Assignee:
            Sid Mahajan
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: