Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-88488

Improve RAM consumption for trial runs of cached plans in SBE

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 8.0.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • Fully Compatible
    • QE 2024-04-01

      In classic runtime planner for SBE when running a trial period for cached plan, we track the amount of documents that are returned by find part of the query, not the whole pipeline.

      Consider the pipeline: 

      [{$match: ..}, {$lookup: ..}, {$unwind: ..}]
      

      When first we multi-plan the query, we will run only $match part.

      After that we will extend the plan with the pipeline and run it.

      In SBE plan cache, we will only have the whole plan.

       

      When recovering a plan from cache, we want to run a trial period to make sure plan is still effective enough. So we set a read budget and run the plan until it returns enough documents.

      We can't just look at the amount of documents, returned by the whole plan, as $unwind may affect it randomly.

      So we track via TrialRunTracker the amount of documents, returned by the part of the plan that correspond to $match stage only.

      However, we can't run only a part of the plan, so we run the whole plan. 

      And when the whole plan returns a document, we need to stash it.

      However, $lookup + $unwind can inflate the amount of memory needed for this stash.

      Currently we just cap the stash at 101 * max document size.

      We want to come up with more memory efficient way to do with.

       

            Assignee:
            ivan.fefer@mongodb.com Ivan Fefer
            Reporter:
            ivan.fefer@mongodb.com Ivan Fefer
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: