Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-57667

Improve processing speed for resharding's collection cloning pipeline

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0.3, 5.1.0-rc0
    • Component/s: Query Language
    • Labels:
      None
    • Backwards Compatibility:
      Minor Change
    • Backport Requested:
      v5.0
    • Sprint:
      Query Optimization 2021-07-12, Query Optimization 2021-07-26, Query Optimization 2021-08-09
    • Linked BF Score:
      170

      Description

      SERVER-57483 partially addressed the regression of $lookup results not being cached by special casing the config.cache.chunks collection. There is still room for significant improvement. It has been found that using ChunkManager to binary search the temporary resharding collection's chunk ranges rather than using DocumentSourceSequentialDocumentCache to scan linearly through the same chunk ranges results in a >10x speedup in the overall collection cloning runtime.

      This ticket represents the work to add a custom aggregation stage or expression or otherwise optimize resharding's collection cloning pipeline to achieve the observed >10x speedup.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              joshua.lapacik Joshua Lapacik
              Reporter:
              max.hirschhorn Max Hirschhorn
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: