Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35894

Create a $exchange producer stage

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.1
    • Component/s: Aggregation Framework
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Epic Link:
    • Sprint:
      Query 2018-07-02, Query 2018-07-16

      Description

      As part of the work to improve the $out stage, we would like to work towards a model where multiple shards can perform the "merger" work for the pipeline in parallel. Specifically, when we are outputting to a sharded collection from a sharded collection, we would like each shard to partition the output data to each other shard based on the output shard key. This is in contrast to our current sharded execution strategy, which would gather all the results from each shard to one "merger" process, which would then have to scatter the writes across the cluster.

      We believe the "exchange" operator/model would be useful here, and that the AsyncResultsMerger fits pretty well as the consumer part of the exchange. We also believe that the TeeBuffer class is pretty close to acting as a producer, but needs to be remodeled and adapted to serve requests from multiple threads.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              martin.neupauer Martin Neupauer
              Reporter:
              charlie.swanson Charlie Swanson
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: