Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2115

fast-delete pages can be incorrectly lost due to a crash

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:
      None
    • # Replies:
      12
    • Last comment by Customer:
      true

      Description

      Michael Cahill, following up on this GitHub conversation

      I was asking the question badly then, but I think there's a real problem.

      I have a test case (attached) which does this:

      • Create a single table, populate it with 1000 key/value pairs.
      • Close and re-open the database, so we can fast-delete pages.
      • Truncate a chunk of the key/value pairs inside a transaction.
      • With the truncation uncommitted, checkpoint the database.
      • Crash.
      • Open the database
      • Check that all the keys are there.

      The output I get is:

      truncate 290 to 500
      key 294, WT_NOTFOUND: item not found
      ret: -31803 (WT_NOTFOUND: item not found)
      Assertion failed: (__r == 0), function main, file t2.c, line 45.
      Abort trap (core dumped)
      

      What's going on is I'm truncating key/value pairs 290 to 500, and the first key on a new page boundary is key 294, so that page is fast-deleted, and after the crash, that's the first key we don't see.

      The problem is not that we don't write the backing leaf page correctly (I think we do), but the internal page has a cell type of WT_CELL_DEL and when we read it, we assume all of the keys on the page have been deleted, which isn't correct because the deleting transaction never committed.

      I think we need to fix the code in reconciliation to not write WT_CELL_DEL cells unless the delete is globally visible, but I'd need to stare at the code some more to be sure.

      And, we need to think about named checkpoints, specifically in the page-reading code, where we recently made the change to short-circuit any deleted page or look-aside table handling if we're reading from a checkpoint handle – I think it's OK, but someone needs to review in the context of this ticket.

      And, maybe there are logging implications?

      Let me know if I'm just missing something!

      1. WT2115.tar.gz
        1 kB
        Keith Bostic

        Issue Links

          Activity

          Hide
          michael.cahill Michael Cahill added a comment - - edited

          Keith Bostic, the Python code you're looking for is helper.copy_wiredtiger_home, e.g., in test_durability01.py.

          I thought I'd convinced myself that this shouldn't matter for MongoDB capped collections, but I'd be more comfortable if we backported this one to avoid having to debug at some point in the future in the field.

          Show
          michael.cahill Michael Cahill added a comment - - edited Keith Bostic , the Python code you're looking for is helper.copy_wiredtiger_home , e.g., in test_durability01.py . I thought I'd convinced myself that this shouldn't matter for MongoDB capped collections, but I'd be more comfortable if we backported this one to avoid having to debug at some point in the future in the field.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: WT-2115: add a test for the bug.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/e8a1796428ada400cbbc623767797eb956f0d789

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: WT-2115 : add a test for the bug. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e8a1796428ada400cbbc623767797eb956f0d789
          Hide
          keith.bostic Keith Bostic added a comment -

          Michael Cahill, I've added the static test for WT-2115, so the only remaining task is the backport, happy to do that if you tell me in what branch you want it done.

          Show
          keith.bostic Keith Bostic added a comment - Michael Cahill , I've added the static test for WT-2115 , so the only remaining task is the backport, happy to do that if you tell me in what branch you want it done.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2115 Don't skip truncated pages that are part of a checkpoint

          Merge pull request #2206 from wiredtiger/WT-2115

          (cherry picked from commit 10c2f15b087327ab0ce0a790b4d1266dec366541)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/bbc2d3efc8bc8e8581df632a8970ff2a4497b161

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2115 Don't skip truncated pages that are part of a checkpoint Merge pull request #2206 from wiredtiger/ WT-2115 (cherry picked from commit 10c2f15b087327ab0ce0a790b4d1266dec366541) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/bbc2d3efc8bc8e8581df632a8970ff2a4497b161
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2115 Don't skip truncated pages that are part of a checkpoint

          Merge pull request #2206 from wiredtiger/WT-2115

          (cherry picked from commit 10c2f15b087327ab0ce0a790b4d1266dec366541)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/bbc2d3efc8bc8e8581df632a8970ff2a4497b161

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2115 Don't skip truncated pages that are part of a checkpoint Merge pull request #2206 from wiredtiger/ WT-2115 (cherry picked from commit 10c2f15b087327ab0ce0a790b4d1266dec366541) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/bbc2d3efc8bc8e8581df632a8970ff2a4497b161

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 30 weeks, 4 days ago
                Date of 1st Reply: