Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9840

Verify fails after salvage of FLCS

    • 3
    • 2023-06-27 Lord of the Sprints

      This failure is a format run with ops.salvage=1. After format performs a salvage operation, it verifies the resulting table, to confirm that salvage reconstructed a correct tree. In this case, apparently, it didn't as the verify failed.

      Failure messages:

      [1662446542:405936][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_dsk_col_fix, 817: last byte of data on page at [0: 80896-111616, 30720, 2966842036] contains trailing garbage
      [1662446542:405978][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 520: checkpoint ranges never verified: 3
      [1662446542:405984][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_filefrag_chk, 434: file ranges never verified: 2
      t: FAILED: table_verify/49: ret == 0 || ret == EBUSY

      Stack trace:

      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x0000ffff909faaac in __GI_abort () at abort.c:79
      #2  0x0000ffff90d82240 in __wt_abort (session=session@entry=0x0) at ../src/os_common/os_abort.c:30
      #3  0x000000000041b2d8 in testutil_die (e=e@entry=0, fmt=fmt@entry=0x41c270 "%s/%d: %s") at ../test/utility/misc.c:74
      #4  0x00000000004169c8 in table_verify (table=table@entry=0x21740000, arg=<optimized out>) at ../test/format/verify.c:49
      #5  0x0000000000412568 in wts_salvage (table=0x21740000, arg=<optimized out>) at ../test/format/salvage.c:156
      #6  0x0000000000414164 in tables_apply (func=0x412398 <wts_salvage>, arg=arg@entry=0x0) at ../test/format/format.i:182
      #7  0x0000000000414bb0 in main (argc=<optimized out>, argv=<optimized out>) at ../test/format/t.c:395

      CONFIG:

      ############################################
      #  RUN PARAMETERS: V3
      ############################################
      assert.read_timestamp=0
      backup=0
      backup.incremental=off
      backup.incr_granularity=1208
      block_cache=0
      block_cache.cache_on_checkpoint=0
      block_cache.cache_on_writes=1
      block_cache.size=44
      btree.huffman_value=0
      cache=594
      cache.evict_max=1
      cache.minimum=20
      checkpoint=on
      checkpoint.log_size=25
      checkpoint.wait=25
      disk.data_extend=0
      disk.direct_io=0
      disk.encryption=none
      disk.mmap=1
      disk.mmap_all=0
      format.abort=0
      format.independent_thread_rng=1
      format.major_timeout=0
      import=0
      logging=0
      logging.compression=none
      logging.file_max=477965
      logging.prealloc=1
      logging.remove=0
      ops.alter=0
      ops.compaction=0
      ops.hs_cursor=1
      ops.prepare=0
      ops.random_cursor=0
      ops.salvage=1
      ops.verify=1
      quiet=1
      runs.in_memory=0
      runs.ops=0
      runs.rows=1800000
      runs.tables=3
      runs.threads=32
      runs.timer=16
      runs.verify_failure_dump=0
      statistics=0
      statistics.server=0
      stress.aggressive_sweep=0
      stress.checkpoint=0
      stress.checkpoint_evict_page=0
      stress.checkpoint_reserved_txnid_delay=0
      stress.checkpoint_prepare=0
      stress.evict_reposition=0
      stress.failpoint_eviction_fail_after_reconciliation=0
      stress.failpoint_hs_delete_key_from_ts=0
      stress.hs_checkpoint_delay=0
      stress.hs_search=0
      stress.hs_sweep=0
      stress.split_1=0
      stress.split_2=0
      stress.split_3=0
      stress.split_4=1
      stress.split_5=0
      stress.split_6=0
      stress.split_7=0
      transaction.implicit=70
      transaction.timestamps=0
      wiredtiger.config=off
      wiredtiger.rwlock=1
      wiredtiger.leak_memory=0
      ############################################
      #  TABLE PARAMETERS: table 1
      ############################################
      table1.btree.bitcnt=5
      table1.btree.compression=snappy
      table1.btree.internal_key_truncation=1
      table1.btree.internal_page_max=15
      table1.btree.leaf_page_max=17
      table1.btree.memory_page_max=4
      table1.btree.split_pct=97
      table1.disk.checksum=unencrypted
      table1.disk.firstfit=0
      table1.ops.pct.delete=3
      table1.ops.pct.insert=66
      table1.ops.pct.modify=31
      table1.ops.pct.read=0
      table1.ops.pct.write=0
      table1.ops.truncate=1
      table1.runs.mirror=0
      table1.runs.source=table
      table1.runs.type=fixed-length column-store
      ############################################
      #  TABLE PARAMETERS: table 2
      ############################################
      table2.btree.compression=zlib
      table2.btree.dictionary=0
      table2.btree.internal_key_truncation=1
      table2.btree.internal_page_max=13
      table2.btree.leaf_page_max=9
      table2.btree.memory_page_max=1
      table2.btree.repeat_data_pct=81
      table2.btree.split_pct=62
      table2.btree.value_max=433
      table2.btree.value_min=8
      table2.disk.checksum=unencrypted
      table2.disk.firstfit=1
      table2.ops.pct.delete=0
      table2.ops.pct.insert=1
      table2.ops.pct.modify=62
      table2.ops.pct.read=36
      table2.ops.pct.write=1
      table2.ops.truncate=1
      table2.runs.mirror=0
      table2.runs.source=table
      table2.runs.type=variable-length column-store
      ############################################
      #  TABLE PARAMETERS: table 3
      ############################################
      table3.btree.bitcnt=7
      table3.btree.compression=lz4
      table3.btree.internal_key_truncation=1
      table3.btree.internal_page_max=13
      table3.btree.leaf_page_max=17
      table3.btree.memory_page_max=4
      table3.btree.split_pct=84
      table3.disk.checksum=off
      table3.disk.firstfit=1
      table3.ops.pct.delete=22
      table3.ops.pct.insert=71
      table3.ops.pct.modify=1
      table3.ops.pct.read=0
      table3.ops.pct.write=6
      table3.ops.truncate=1
      table3.runs.mirror=0
      table3.runs.source=table
      table3.runs.type=fixed-length column-store

       

      format-stress-test-4 on ubuntu2004-stress-tests-arm64

      Host: i-0c0a09f5e7262b1fb
      Project: wiredtiger
      Commit: c4fa0f75

      Task Logs:

      format-stress-test-4 task_log

      Logs:

      Verbose Error message
          [1662446542:405936][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_dsk_col_fix, 817: last byte of data on page at [0: 80896-111616, 30720, 2966842036] contains trailing garbage
          [1662446542:405978][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_ckptfrag_chk, 520: checkpoint ranges never verified: 3
          [1662446542:405984][1192542:0xffff9111d480], t, file:T00003.wt, WT_SESSION.verify: [WT_VERB_DEFAULT][ERROR]: __verify_filefrag_chk, 434: file ranges never verified: 2
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_arm64_format_stress_test_4_c4fa0f7511202f4b9908e0b70a444197dddd07c2_22_09_06_01_38_34/0?type=T#L1308
      
      C/CPP Suite Failure message
          t: FAILED: table_verify/49: ret == 0 || ret == EBUSY
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_arm64_format_stress_test_4_c4fa0f7511202f4b9908e0b70a444197dddd07c2_22_09_06_01_38_34/0?type=T#L1311
      
      Failure keywords
          t: process aborting
          WiredTiger Error: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_arm64_format_stress_test_4_c4fa0f7511202f4b9908e0b70a444197dddd07c2_22_09_06_01_38_34/0?type=T#L1451
      
      Evergreen YAML Script Failure
      Command failed: shell script encountered problem: exit code 1
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_arm64_format_stress_test_4_c4fa0f7511202f4b9908e0b70a444197dddd07c2_22_09_06_01_38_34/0?type=T#L1849
      

      logs

            Assignee:
            ravi.giri@mongodb.com Ravi Giri
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: