Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1745

Allow dirty eviction per file as checkpoint progresses

    Details

    • Type: Task
    • Status: Closed
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      At the moment the transaction ID for a checkpoint causes us to hold all of the updates required to service that transaction in memory (i.e: blocks eviction). In a workload that consists of many short transactions and regular checkpoints those updates can be unnecessary.

      I'm planning to implement a scheme were:

      • We track a checkpoint generation.
      • Each file will reference it's current checkpoint generation. Once a checkpoint has finished in a particular file it will bump it's checkpoint generation.
      • We track the oldest transaction ID excluding the checkpoint ID, and the checkpoint ID.

      When checking whether a page can be evicted:

      • If checkpoint generation is equal to the current checkpoint generation: Use oldest ID.
      • If checkpoint generation is before the current checkpoint generation: Use older of oldest ID and checkpoint ID.

      Use a similar scheme in the obsolete update check of

      .
       
      All other places in the code use the oldest ID as a visibility check. To implement the visibility changes I'll likely split 

      __wt_txn_visible_all{{`}} into two variations - one as it currently exists and another that includes the checkpoint ID.

        Issue Links

          Activity

          Hide
          ramon.fernandez Ramon Fernandez added a comment -
          Show
          ramon.fernandez Ramon Fernandez added a comment - Additional ticket information from GitHub This ticket was referenced in the following commits: 272daddb3734cd196cd303df4271f7e9e8f00cd3 4d0b4093e3d3c9fa0be2bbb01467579f05deddc5 7ee20176acc77aef2c74eec2e4259b9929fb6dc5 110218b40844618b343301a70355003652e4be61
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Export a function to get the oldest transaction ID that needs to be tracked for a given tree. Managing cached overflow items was using the oldest transaction ID, but the meaning of that field changed.

          This fixes a bug that was introduced by WT-1745, that requires a combination of many overflow values, eviction and checkpoints, but can lead to unexpected WT_NOTFOUND errors during reconciliation.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/98786589218e4d3f06232633e5be6a7e619fa165

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Export a function to get the oldest transaction ID that needs to be tracked for a given tree. Managing cached overflow items was using the oldest transaction ID, but the meaning of that field changed. This fixes a bug that was introduced by WT-1745 , that requires a combination of many overflow values, eviction and checkpoints, but can lead to unexpected WT_NOTFOUND errors during reconciliation. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/98786589218e4d3f06232633e5be6a7e619fa165

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 44 weeks, 6 days ago
                Date of 1st Reply: