disagg switch format run more than 15 minutes past the maximum time

XMLWordPrintableJSON

    • Type: Build Failure
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction

      format-stress-test-disagg-switch-data-validation-2 on ubuntu2004-asan

      Host: i-02be74a159a59884d
      Project: wiredtiger
      Commit: d76c75aa
      Please refer to BF(G) Playbook for instructions on handling BF and BFG tickets as well as Auto-Resolution Rules

      Task Logs:

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/products.sh: line 13: /opt/mongodbtoolchain/revisions/93d85cc1a00ca1d53fcc7d4ca790f19a4e5dd542/scripts/installer.sh: No such file or directory
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      [450/680] Building C object test/csuite/CMakeFiles/test_wt16990_disagg_checkpoint_panic.dir/wt16990_disagg_checkpoint_panic/main.c.o
      [462/680] Linking CXX executable test/csuite/wt16990_disagg_checkpoint_panic/test_wt16990_disagg_checkpoint_panic
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      format run more than 15 minutes past the maximum time
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      format run dumping cache and transaction state, then aborting the process
      

      logs

      format-stress-test-disagg-switch-data-validation-2 task_log

      Logs:

      0x7fd0ba34c800:transaction state dump
      0x7fd0ba34c800:current ID: 2033122
      0x7fd0ba34c800:last running ID: 1974661
      0x7fd0ba34c800:metadata_pinned ID: 1971366
      0x7fd0ba34c800:oldest ID: 1971366
      0x7fd0ba34c800:durable timestamp: (0, 7943393)
      0x7fd0ba34c800:oldest timestamp: (0, 7889634)
      0x7fd0ba34c800:pinned timestamp: (0, 7889634)
      0x7fd0ba34c800:stable timestamp: (0, 7889634)
      0x7fd0ba34c800:stable disaggregated schema epoch: (0, 0)
      0x7fd0ba34c800:has_durable_timestamp: yes
      0x7fd0ba34c800:has_oldest_timestamp: yes
      0x7fd0ba34c800:has_pinned_timestamp: yes
      0x7fd0ba34c800:has_stable_timestamp: yes
      0x7fd0ba34c800:has_stable_disaggregated_schema_epoch: no
      0x7fd0ba34c800:oldest_is_pinned: yes
      0x7fd0ba34c800:stable_is_pinned: yes
      0x7fd0ba34c800:checkpoint running: no
      0x7fd0ba34c800:checkpoint generation: 15
      0x7fd0ba34c800:checkpoint pinned ID: 0
      0x7fd0ba34c800:checkpoint txn ID: 0
      0x7fd0ba34c800:session count: 32
      0x7fd0ba34c800:Transaction state of active sessions:
      0x7fd0ba34c800:session ID: 14, txn ID: 1974661, pinned ID: 1971366, metadata pinned ID: 0, name: WT_CURSOR.next
      0x7fd0ba34c800:transaction id: 1974661, mod count: 16732, snap min: 1971366, snap max: 1974515, snapshot count: 9, snapshot: [1971366, 1974494, 1974497, 1974498, 1974502, 1974503, 1974507, 1974510, 1974513], commit_timestamp: (0, 0), durable_timestamp: (0, 0), first_commit_timestamp: (0, 0), prepare_timestamp: (0, 0), prepared id: 0, pinned_durable_timestamp: (0, 0), read_timestamp: (0, 0), checkpoint LSN: [0,0], full checkpoint: false, flags: 0x00010204, isolation: WT_ISO_SNAPSHOT, last saved error code: 0, last saved sub-level error code: -32000, last saved error message: last API call was successful
      0x7fd0ba34c800:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7fd0ba34c800:cache dump
      0x7fd0ba34c800:cache full: no
      0x7fd0ba34c800:cache clean check: no (66.079%)
      0x7fd0ba34c800:cache dirty check: no (12.571%)
      0x7fd0ba34c800:cache updates check: no (6.434%)
      0x7fd0ba34c800:file:T00003.wt_stable/WiredTigerCheckpoint.35(<live>):
      0x7fd0ba34c800:internal: 17 pages, 351.69 KB, 17/0 clean/dirty pages, 351.69/0.00 clean / dirty KB, 82.24 KB max page, 0.00 KB max dirty page
      0x7fd0ba34c800:leaf: 332 pages, 10620.61 KB, 332/0 clean/dirty pages, 10620.61 /0.00 /0.00 clean/dirty/updates KB, 179.50 KB max page, 0.00 KB max dirty page
      0x7fd0ba34c800:file:WiredTigerSharedHS.wt_stable/WiredTigerCheckpoint.27(<live>):
      0x7fd0ba34c800:internal: 1 pages, 5.41 KB, 1/0 clean/dirty pages, 5.41/0.00 clean / dirty KB, 5.41 KB max page, 0.00 KB max dirty page
      0x7fd0ba34c800:leaf: 0 pages
      0x7fd0ba34c800:file:T00002.wt_stable/WiredTigerCheckpoint.35(<live>):
      AddressSanitizer:DEADLYSIGNAL
      =================================================================
      ==33276==ERROR: AddressSanitizer: SEGV on unknown address 0x0000000000d8 (pc 0x7fd0bb195c8f bp 0x7ffd354f9680 sp 0x7ffd354f8ea0 T0)
      ==33276==The signal is caused by a READ memory access.
      ==33276==Hint: address points to the zero page.
          #0 0x7fd0bb195c8f in __wt_page_in_func /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/btree/bt_read.c:553:13
          #1 0x7fd0bb2b62ba in __wt_page_swap_func /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/include/btree_inline.h:2703:11
          #2 0x7fd0bb2b39d0 in __tree_walk_internal /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/btree/bt_walk.c:329:19
          #3 0x7fd0bb2b258e in __wt_tree_walk /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/btree/bt_walk.c:413:13
          #4 0x7fd0bb6d6b50 in __verbose_dump_cache_single /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/evict/evict_verbose.c:62:12
          #5 0x7fd0bb6d5ce3 in __verbose_dump_cache_apply /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/evict/evict_verbose.c:152:9
          #6 0x7fd0bb6d4b1b in __wt_verbose_dump_cache /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/evict/evict_verbose.c:193:5
          #7 0x7fd0bb3ccfd5 in __conn_debug_info /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/conn/conn_api.c:1437:9
          #8 0x5555f753f644 in operations /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/test/format/ops.c:506:19
          #9 0x5555f7567471 in main /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/test/format/t.c:411:13
          #10 0x7fd0bab4a082 in __libc_start_main /build/glibc-FcRMwW/glibc-2.31/csu/../csu/libc-start.c:308:16
          #11 0x5555f750ba8d in _start (/data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/cmake_build/test/format/t+0x31a8d)
      ==33276==Register values:
      rax = 0x00000000000000d8  rbx = 0x00007ffd354f8ea0  rcx = 0xf2f2f2f2f2f2f2f2  rdx = 0x0000000000000080
      rdi = 0x00000ffa9711bea4  rsi = 0x00000000000000f8  rbp = 0x00007ffd354f9680  rsp = 0x00007ffd354f8ea0
       r8 = 0x00007fffffffff01   r9 = 0x0000000000000001  r10 = 0x00007fffffffff01  r11 = 0x4000000000000001
      r12 = 0x00005555f750ba60  r13 = 0x00007ffd354fb710  r14 = 0x0000000000000000  r15 = 0x0000000000000000
      AddressSanitizer can not provide additional info.
      SUMMARY: AddressSanitizer: SEGV /data/mci/b35723b818eca2caa7e73e04e218663f/wiredtiger/src/btree/bt_read.c:553:13 in __wt_page_in_func
      ==33276==ABORTING
      AddressSanitizer:DEADLYSIGNAL
      AddressSanitizer:DEADLYSIGNAL
      AddressSanitizer:DEADLYSIGNAL
      AddressSanitizer:DEADLYSIGNAL
      bash: line 68: 33276 Aborted                 (core dumped) ./t -R $format_args
      ======= FAILURE ==========
      ############################################
      #  RUN PARAMETERS: V3
      ############################################
      assert.read_timestamp=0
      background_compact=0
      background_compact.free_space_target=71
      backup=0
      backup.incremental=off
      backup.incr_granularity=4661
      backup.live_restore=0
      backup.live_restore_read_size=4
      backup.live_restore_threads=8
      block_cache=0
      block_cache.cache_on_checkpoint=0
      block_cache.cache_on_writes=1
      block_cache.size=1
      cache=3072
      cache.evict_max=3
      cache.eviction_dirty_target=0
      cache.eviction_dirty_trigger=0
      cache.eviction_updates_target=0
      cache.eviction_updates_trigger=0
      cache.minimum=0
      cache.maximum=0
      checkpoint=on
      checkpoint.log_size=146
      checkpoint.wait=13
      checkpoint_threads=4
      compact.free_space_target=85
      debug.background_compact=0
      debug.checkpoint_retention=9
      debug.cursor_reposition=1
      debug.disagg_slow_truncate_follower=1
      debug.eviction=0
      debug.log_retention=0
      debug.realloc_exact=0
      debug.realloc_malloc=0
      debug.slow_checkpoint=0
      debug.slow_truncate=0
      debug.table_logging=0
      debug.update_restore_evict=0
      disagg.internal_page_delta=1
      disagg.leaf_page_delta=1
      disagg.multi=0
      disagg.multi_validation=0
      disagg.enabled=1
      disagg.layered=1
      disagg.mode=switch
      disagg.page_log=palite
      disagg.key_provider=1
      disagg.page_log.verbose=0
      disagg.drain_threads=15
      disagg.preserve=0
      disagg.stepdown_async=0
      disk.data_extend=0
      disk.encryption=none
      disk.mmap=1
      disk.mmap_all=0
      eviction.evict_use_softptr=0
      file_manager.close_handle_minimum=29
      file_manager.close_idle_time=16
      file_manager.close_scan_interval=8
      format.abort=0
      format.independent_thread_rng=1
      format.major_timeout=0
      import=0
      logging=0
      logging.compression=none
      logging.file_max=254544
      logging.prealloc=1
      logging.remove=1
      obsolete_cleanup.method=off
      obsolete_cleanup.wait=1413
      ops.alter=0
      ops.compaction=0
      ops.hs_cursor=0
      ops.bound_cursor=0
      ops.prepare=0
      ops.reserve=3
      ops.random_cursor=0
      ops.salvage=0
      ops.throttle=0
      ops.throttle.sleep_us=587420
      ops.verify=1
      prefetch=0
      prefetch.default=0
      precise_checkpoint=1
      preserve_prepared=0
      quiet=1
      random.data_seed=1268043
      random.extra_seed=4854945
      rollback_to_stable_threads=4
      runs.in_memory=0
      runs.mirror=1
      runs.ops=0
      runs.predictable_replay=0
      runs.source=layered
      runs.tables=3
      runs.threads=16
      runs.timer=7
      runs.type=row-store
      runs.verify_failure_dump=0
      statistics.mode=fast
      statistics_log.sources=off
      stress.aggressive_stash_free=1
      stress.aggressive_sweep=0
      stress.checkpoint=0
      stress.checkpoint_evict_page=0
      stress.checkpoint_prepare=0
      stress.compact_slow=0
      stress.evict_reposition=1
      stress.failpoint_eviction_split=1
      stress.failpoint_hs_delete_key_from_ts=0
      stress.failpoint_rec_before_wrapup=0
      stress.hs_checkpoint_delay=0
      stress.hs_search=0
      stress.hs_sweep=0
      stress.prefetch_delay=0
      stress.prepare_resolution_1=1
      stress.sleep_before_read_overflow_onpage=0
      stress.split_1=0
      stress.split_2=0
      stress.split_3=0
      stress.split_4=0
      stress.split_5=0
      stress.split_6=0
      stress.split_7=0
      stress.split_8=0
      tiered_storage.flush_frequency=0
      tiered_storage.storage_source=off
      transaction.implicit=0
      transaction.operation_timeout_ms=2000
      transaction.timestamps=1
      wiredtiger.config=off
      wiredtiger.rwlock=1
      wiredtiger.leak_memory=0
      ############################################
      #  TABLE PARAMETERS: table 1
      ############################################
      table1.btree.compression=none
      table1.btree.dictionary=0
      table1.btree.internal_key_truncation=1
      table1.btree.internal_page_max=9
      table1.btree.key_max=117
      table1.btree.key_min=18
      table1.btree.leaf_page_max=14
      table1.btree.memory_page_max=3
      table1.btree.prefix_len=0
      table1.btree.prefix_compression=1
      table1.btree.prefix_compression_min=3
      table1.btree.reverse=0
      table1.btree.split_pct=69
      table1.btree.value_max=1443
      table1.btree.value_min=20
      table1.disagg.enabled=0
      table1.disk.checksum=unencrypted
      table1.disk.firstfit=0
      table1.ops.pareto=0
      table1.ops.pareto.skew=31
      table1.ops.pct.delete=6
      table1.ops.pct.insert=60
      table1.ops.pct.modify=25
      table1.ops.pct.read=4
      table1.ops.pct.write=5
      table1.ops.truncate=1
      table1.runs.rows=984811
      table1.runs.source=table
      ############################################
      #  TABLE PARAMETERS: table 2
      ############################################
      table2.btree.compression=none
      table2.btree.dictionary=0
      table2.btree.internal_key_truncation=1
      table2.btree.internal_page_max=15
      table2.btree.key_max=32
      table2.btree.key_min=20
      table2.btree.leaf_page_max=16
      table2.btree.memory_page_max=3
      table2.btree.prefix_len=0
      table2.btree.prefix_compression=0
      table2.btree.prefix_compression_min=0
      table2.btree.reverse=0
      table2.btree.split_pct=94
      table2.btree.value_max=3198
      table2.btree.value_min=15
      table2.disk.checksum=on
      table2.disk.firstfit=0
      table2.ops.pareto=0
      table2.ops.pareto.skew=64
      table2.ops.pct
      

      logs

      Repro Artifacts:

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: