Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2119

evicting multi-block pages with overflow keys can corrupt checkpoints

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      20
    • Last comment by Customer:
      true

      Description

      The error output was:

      1: table, row-store (500 seconds)
      + eval nice ./t file_type=row data_source=file leaf_page_max=9 internal_page_max=9 key_min=256 value_min=256 cache=10
      ++ nice ./t file_type=row data_source=file leaf_page_max=9 internal_page_max=9 key_min=256 value_min=256 cache=10
      [1442802715:432390][5332:0x7faa96a45780], t, file:wt, WT_SESSION.verify: fragment at 103251456 referenced multiple times in a single checkpoint or found in the checkpoint but not listed in the checkpoint's allocation list: WT_ERROR: non-specific WiredTiger error
      [1442802715:432454][5332:0x7faa96a45780], t, file:wt, WT_SESSION.verify: cell 148 on page at [1629696-1633280, 3584, 3136215260] references an overflow item at [103251456-103251968, 512, 2000602526] that failed verification: WT_ERROR: non-specific WiredTiger error
      [1442802717:394401][5332:0x7faa96a45780], t, file:wt, WT_SESSION.verify: checkpoint ranges never verified: 9861
      [1442802719:712282][5332:0x7faa96a45780], t, file:wt, WT_SESSION.verify: file ranges never verified: 13294
      t: session.verify: file:wt: post-ops verify: WT_ERROR: non-specific WiredTiger error
      

      The build that failed is:

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/7065/console

      test/format configuration file:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      auto_throttle=1
      firstfit=0
      bitcnt=2
      bloom=1
      bloom_bit_count=8
      bloom_hash_count=4
      bloom_oldest=0
      cache=10
      checkpoints=1
      checksum=uncompressed
      chunk_size=8
      compaction=0
      compression=bzip-raw
      data_extend=0
      data_source=file
      delete_pct=39
      dictionary=0
      encryption=none
      evict_max=4
      file_type=row-store
      backups=0
      huffman_key=0
      huffman_value=0
      insert_pct=8
      internal_key_truncation=1
      internal_page_max=9
      isolation=read-uncommitted
      key_gap=18
      key_max=256
      key_min=256
      leak_memory=0
      leaf_page_max=9
      logging=0
      logging_compression=lz4
      logging_archive=1
      logging_prealloc=1
      long_running_txn=0
      lsm_worker_threads=3
      merge_max=14
      mmap=0
      ops=100000
      prefix_compression=1
      prefix_compression_min=2
      repeat_data_pct=9
      reverse=1
      rows=100000
      runs=1
      salvage=1
      split_pct=71
      statistics=0
      statistics_server=0
      threads=6
      timer=20
      value_max=2039
      value_min=256
      verify=1
      wiredtiger_config=
      write_pct=78
      ############################################
      

        Issue Links

          Activity

          Hide
          keith.bostic Keith Bostic added a comment -

          Michael Cahill, I think I understand this one.

          It doesn't matter if the page being reconciled has an overflow item on it, it matters if the page's parent's key (the key on the parent that references the split page), is an overflow item. Unfortunately, this is complicated by the split_parent code: we opportunistically discard deleted WT_REF items from the parent's page-index array, as well as deepen the tree if the parent is getting too large, and both of those actions discard WT_REF items from the parent, and potentially discard overflow keys.

          I did a branch that mostly works, but it's very complicated, it's relatively tricky to figure out if the key for a random WT_REF is an overflow item.

          Show
          keith.bostic Keith Bostic added a comment - Michael Cahill , I think I understand this one. It doesn't matter if the page being reconciled has an overflow item on it, it matters if the page's parent's key (the key on the parent that references the split page), is an overflow item. Unfortunately, this is complicated by the split_parent code: we opportunistically discard deleted WT_REF items from the parent's page-index array, as well as deepen the tree if the parent is getting too large, and both of those actions discard WT_REF items from the parent, and potentially discard overflow keys. I did a branch that mostly works, but it's very complicated, it's relatively tricky to figure out if the key for a random WT_REF is an overflow item.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: Revert "WT-2119: We can't evict clean, multiblock row-store pages with overflow"

          This reverts commit 74b31ef9a7157b01b275722f6bb86bb918fc47d7.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/5e36773217c903d49118536586c1a8df5dc99f83

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Revert " WT-2119 : We can't evict clean, multiblock row-store pages with overflow" This reverts commit 74b31ef9a7157b01b275722f6bb86bb918fc47d7. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/5e36773217c903d49118536586c1a8df5dc99f83
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: WT-2119: Add a new WT_PAGE flag, WT_PAGE_OVERFLOW_KEYS, set on any
          internal page with overflow keys, use it to ensure we never evict a
          multi-block child page with a parent that has overflow keys during
          a checkpoint.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/dd6fde79047dba5894ffa5e5b62f637f89a6b9ba

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2119 : Add a new WT_PAGE flag, WT_PAGE_OVERFLOW_KEYS, set on any internal page with overflow keys, use it to ensure we never evict a multi-block child page with a parent that has overflow keys during a checkpoint. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/dd6fde79047dba5894ffa5e5b62f637f89a6b9ba
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Merge pull request #2219 from wiredtiger/wt2119

          WT-2119: don't evict clean multiblock pages with overflow items during checkpoints.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/51cf672993aac58931372fa5a81e9e95af3970f6

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Merge pull request #2219 from wiredtiger/wt2119 WT-2119 : don't evict clean multiblock pages with overflow items during checkpoints. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/51cf672993aac58931372fa5a81e9e95af3970f6
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: WT-2119: Forgot to add WT_PAGE_OVERFLOW_KEYS to the list of page-flags
          in the debug page display function.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/b6aac198b325fa13bb7118cd811187c6ecbd0566

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2119 : Forgot to add WT_PAGE_OVERFLOW_KEYS to the list of page-flags in the debug page display function. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/b6aac198b325fa13bb7118cd811187c6ecbd0566

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 30 weeks, 5 days ago
                Date of 1st Reply: