Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-57062

Improve memory footprint of document/value caching in window functions

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization

      As part of the commit to account for freed documents when they fall out of a window, we made an unfortunate compromise to handle the inflation of the internal Document cache by forcing all fields to be cached up front. This is not efficient and may end up forcing a spill to disk more often than necessary. Several options discussed to avoid doing this for every document:

      • Use the required fields from the dependency analysis to only populate the Document cache for the fields we need. Note that if a required field lands on an object, some expressions (e.g. $max) will recurse into the nested object to do a field-by-field comparison.
      • Detect the changing in memory footprint on each expression evaluation. This one is tedious and likely to break if we were to add more functions or window types.
      • Temporarily turn off caching in Document while the $setWindowFields stage is holding onto it, since the stage will eventually project a new field it should not affect performance. This could be interesting to try but may be invasive.
      • Add a callback to Document when a field is added to the cache, such that window functions can adjust its memory tracking in near real-time

            Assignee:
            backlog-query-optimization [DO NOT USE] Backlog - Query Optimization
            Reporter:
            nicholas.zolnierz@mongodb.com Nicholas Zolnierz
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: