Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-14453

Investigate changes in SERVER-53762: Report aggregate execution stats in explain for the inner side of $lookup

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 5.0.0-rc0
    • Component/s: manual, Server
    • Labels:
      None
    • Last comment by Customer:
      true
    • Story Points:
      3
    • Sprint:
      ServerDocs2021: Aug17 - Aug24

      Description

      Description

      Downstream Change Summary

      In this ticket, we've modified the explain output for aggregations using $lookup. If the explain verbosity is allPlansExecution or above, we will add total aggregate execution stats to the output. (The specific stats are totalDocsExamined, totalKeysExamined, collectionScans (the number as an integer), and indexesUsed (an array of the index names).

      Description of Linked Ticket

      As part of SERVER-47640, we enhanced the implementation of $lookup to keep track of the PlanSummaryStats associated with the subpipeline. These stats include metrics such as the number of keys examined and the number of documents examined by the subpipeline. Note that the subpipeline may be executed many times, since it is typically executed once per document from the outer (or "local") side. The PlanSummaryStats are tracked in aggregate across all such executions of the subpipeline. So if, for instance, the subpipeline scans a 10 document collection every time it runs, and it runs 5 times, then the result will be a "docsExamined" quantity of 50.

      Now that we are tracking these values, it would be valuable for debugging/diagnostics to expose them in "executionStats" or "allPlansExecution" level explain output. We will need to choose which metrics are exposed, but I think it would at the very least include docs examined and keys examined. This would be especially helpful in understanding the performance properties of the subpipeline, since explain currently does not display the execution plan used on the inner side (see SERVER-22622). Any information in explain as to whether the $lookup subpipeline is cheap or expensive and why would be immensely helpful for query performance debugging scenarios.

      Scope of changes

      Consider if we need to update the $lookup docs as part of this change.

      Impact to Other Docs

      MVP (Work and Date)

      Resources (Scope or Design Docs, Invision, etc.)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jason.price Jason Price
              Reporter:
              backlog-server-pm Backlog - Core Eng Program Management Team
              Participants:
              Last commenter:
              Jason Price Jason Price
              Docs Reviewer:
              Naomi Pentrel Naomi Pentrel
              External Reviewer:
              Mohammad Dashti
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since reply:
                3 weeks, 4 days ago
                Date of 1st Reply: