Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-24978

Second batches in aggregation framework are asked synchronously

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • None
    • 3.7.3
    • Aggregation Framework
    • None
    • Fully Compatible
    • Query 2018-01-29, Query 2018-02-12, Query 2018-02-26, Query 2018-03-12
    • 0

    Description

      In the case of an aggregation framework query the first batch of results asked from the mergerPart to the shards are done asynchronously, however when it needs to get a second batch to fullfill the query it is virtually mono-threaded as the call to getNext is synchronous.

      I think it would be good to be able to overcome this limitation and make all subsequent fetches to be done in the background.

      You can see here (https://github.com/ahom/jupyter-notebooks/blob/master/mongo_cs32044/notebook.ipynb) a representation of this behavior.

      On the Y axis are the shards, on the X axis is the time. Starting from second batch seems is completely synchronous.

      Cheers,
      Antoine

      Attachments

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. SERVER-33660 Once getMores include lsid, sharded aggregations with $mergeCursors can hang

          • Major - P3 - Major loss of function.
          • Closed

          Task - A task that needs to be done. SERVER-32307 Make AsyncResultsMerger kill sequence issue killCursors without waiting for outstanding batches

          • Major - P3 - Major loss of function.
          • Closed
          is depended on by

          Task - A task that needs to be done. SERVER-33280 Add a test to ensure that an error on one shard will proactively clean up any cursors from other shards

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-33323 Refactor $mergeCursors stage to allow it to be used to merge cursors on mongos

          • Major - P3 - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-32014 scanAndOrder serverStatus metric should not be incremented for $sort stages that are merging pre-sorted results

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-24981 $project-$limit optimization has bad repercussion on pipeline splitting

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              charlie.swanson@mongodb.com Charlie Swanson
              antoine.hom@amadeus.com Antoine Hom
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: