Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-57062

Improve memory footprint of document/value caching in window functions

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: None
    • Labels:
      None

      Description

      As part of the commit to account for freed documents when they fall out of a window, we made an unfortunate compromise to handle the inflation of the internal Document cache by forcing all fields to be cached up front. This is not efficient and may end up forcing a spill to disk more often than necessary. Several options discussed to avoid doing this for every document:

      • Use the required fields from the dependency analysis to only populate the Document cache for the fields we need. Note that if a required field lands on an object, some expressions (e.g. $max) will recurse into the nested object to do a field-by-field comparison.
      • Detect the changing in memory footprint on each expression evaluation. This one is tedious and likely to break if we were to add more functions or window types.
      • Temporarily turn off caching in Document while the $setWindowFields stage is holding onto it, since the stage will eventually project a new field it should not affect performance. This could be interesting to try but may be invasive.
      • Add a callback to Document when a field is added to the cache, such that window functions can adjust its memory tracking in near real-time

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              backlog-query-optimization Backlog - Query Optimization
              Reporter:
              nicholas.zolnierz Nicholas Zolnierz
              Participants:
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated: