Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2975

Reduce the memory allocated for statistics

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc3
    • Labels:
      None
    • Sprint:
      Storage 2016-11-21

      Description

      The WiredTiger statistics tracking structure uses 64-bit values, and an array of 23 entries per value to allow for efficiently updating statistics across many threads.

      The consequence of that is that each statistic requires 184 bytes in each dhandle. The dhandle currently maintains approximately 100 statistics, leading to 18k of memory required for each dhandle. Some applications open over 100 thousand handles - which can mean nearly 2GB of memory allocated just for dhandle statistics structures.

      We should consider mechanisms for reducing the memory footprint of our statistics. A possible path forward would be to only keep arrays of values for statistics that are updated across multiple threads in parallel, that's not ideal, since it means identifying that set of statistics (thus potentially adding in performance bottle necks) and will make the statistics tracking macros more complex, but it could save a significant amount of space.

        Attachments

          Activity

            People

            • Assignee:
              keith.bostic Keith Bostic
              Reporter:
              alexander.gorrod Alexander Gorrod
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: