Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2975

Reduce the memory allocated for statistics

    • Type: Icon: Improvement Improvement
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.0, 3.2.11, 3.4.0-rc3
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Storage 2016-11-21

      The WiredTiger statistics tracking structure uses 64-bit values, and an array of 23 entries per value to allow for efficiently updating statistics across many threads.

      The consequence of that is that each statistic requires 184 bytes in each dhandle. The dhandle currently maintains approximately 100 statistics, leading to 18k of memory required for each dhandle. Some applications open over 100 thousand handles - which can mean nearly 2GB of memory allocated just for dhandle statistics structures.

      We should consider mechanisms for reducing the memory footprint of our statistics. A possible path forward would be to only keep arrays of values for statistics that are updated across multiple threads in parallel, that's not ideal, since it means identifying that set of statistics (thus potentially adding in performance bottle necks) and will make the statistics tracking macros more complex, but it could save a significant amount of space.

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: