Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-4107

Reduce lookaside usage in update heavy workloads where history is pinned

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Won't Do
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We have seen workloads that:

      • Do many updates to one or a few large values.
      • Don't allow the oldest transaction/ID or timestamp to move forward.

      Optimizing in such cases is restricted by how WiredTiger manages history - it must be held in cache - we don't version values that are written to data files beyond checkpoints.

      WiredTiger allows more history than fits into cache by using the lookaside (cache overflow) mechanism. It is currently possible to have a page in several different states:

      • On disk (no content in cache).
      • In memory (all history in cache).
      • In "limbo" there is a single version of data in memory, but other history is in the lookaside file.

      Any time an update is made to an entry on a page, the full history needs to be instantiated. In cases where there are hot records the in-memory page image can be greater than the memory_page_max - which is a trigger for WiredTiger forcibly evicting the page.

      We should reproduce a workload that pathologically reads and re-writes pages to lookaside and investigate how to make them more efficient.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              haribabu.kommi Haribabu Kommi
              Reporter:
              alexander.gorrod Alexander Gorrod
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: