Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-1350

Flush dirty pages without evicting them

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.7.0
    • Labels:

      Description

      It would be nice to be able to flush dirty pages from cache without evicting them. Some workloads don't generate a lot of cache pressure, but do want to do checkpoints.

      If the cache is allowed to have a lot of dirty content then checkpoints become more expensive.

      It would be good if code:eviction_dirty_target is set below evicting_target we switched to a mode of only "cleaning" dirty pages, rather than evicting them.

        Issue Links

          Activity

          Hide
          keith.bostic Keith Bostic added a comment -

          Alex, a question about this: isn't that what we do with the initial pass before each checkpoint to clear out dirty pages?

          I guess this would keep checkpoints from taking so long, is that the goal?

          Show
          keith.bostic Keith Bostic added a comment - Alex, a question about this: isn't that what we do with the initial pass before each checkpoint to clear out dirty pages? I guess this would keep checkpoints from taking so long, is that the goal?
          Hide
          keith.bostic Keith Bostic added a comment -

          Alex Gorrod, Michael Cahill, I think I have this working, I was wondering if you had any workloads you thought might be interesting?

          Show
          keith.bostic Keith Bostic added a comment - Alex Gorrod , Michael Cahill , I think I have this working, I was wondering if you had any workloads you thought might be interesting?
          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          There are two things to look out for:

          1) Tracking statistics of how much of the cache is dirty.
          2) Watching for pauses due to the cache getting stuck full during a checkpoint.

          For 1) I'd try running an update intensive wtperf workload. Probably the update-btree.wtperf workload. You may want to increase the cache size, to allow for a reasonable steady state.

          For 2) I'd add in checkpoints to the update-btree.wtperf workload and look for pauses if the cache is running dirty.

          If neither of those workloads are heavy enough to get the cache full, you could try updating the number of worker threads to increase the system load.

          Show
          alexander.gorrod Alexander Gorrod added a comment - There are two things to look out for: 1) Tracking statistics of how much of the cache is dirty. 2) Watching for pauses due to the cache getting stuck full during a checkpoint. For 1) I'd try running an update intensive wtperf workload. Probably the update-btree.wtperf workload. You may want to increase the cache size, to allow for a reasonable steady state. For 2) I'd add in checkpoints to the update-btree.wtperf workload and look for pauses if the cache is running dirty. If neither of those workloads are heavy enough to get the cache full, you could try updating the number of worker threads to increase the system load.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: If the cache is large enough but there are too many dirty pages, don't
          evict the pages, clean them and leave them in the cache; ref WT-1350.

          Abstract out the test for when the cache requires eviction or to reduce
          the number of dirty pages into a new function, __wt_cache_status().
          Change the page eviction function, __wt_evict() to call that function
          after reconciling each page, and leave the page in the cache if the
          cache is currently large enough.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/cff63f7abbe007246bec6e844477e98a39d297c9

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: If the cache is large enough but there are too many dirty pages, don't evict the pages, clean them and leave them in the cache; ref WT-1350 . Abstract out the test for when the cache requires eviction or to reduce the number of dirty pages into a new function, __wt_cache_status(). Change the page eviction function, __wt_evict() to call that function after reconciling each page, and leave the page in the cache if the cache is currently large enough. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/cff63f7abbe007246bec6e844477e98a39d297c9
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'}

          Message: Add an eviction_dirty_trigger configuration setting.

          Allows applications to maintain a low proportion of dirty pages in
          the cache, while allowing clean pages to utilize the cache.

          While here, alter the minimum allowed dirty trigger and target to
          be set as low as 5 percent. Which may be desired for huge caches.

          refs SERVER-16737, WT-1350

          e Please enter the commit message for your changes. Lines starting
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/e075e2c9444549e718a98a345fa967cc92c5da09

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexg@wiredtiger.com'} Message: Add an eviction_dirty_trigger configuration setting. Allows applications to maintain a low proportion of dirty pages in the cache, while allowing clean pages to utilize the cache. While here, alter the minimum allowed dirty trigger and target to be set as low as 5 percent. Which may be desired for huge caches. refs SERVER-16737 , WT-1350 e Please enter the commit message for your changes. Lines starting Branch: develop https://github.com/wiredtiger/wiredtiger/commit/e075e2c9444549e718a98a345fa967cc92c5da09

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 46 weeks, 3 days ago
                Date of 1st Reply: