[SERVER-35894] Create a $exchange producer stage Created: 19/Jun/18  Updated: 29/Oct/23  Resolved: 06/Jul/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 4.1.1

Type: Task Priority: Major - P3
Reporter: Charlie Swanson Assignee: Martin Neupauer
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-35940 Remove IntrusiveCounter in favor of R... Closed
is depended on by SERVER-36275 Support sending non-contiguous ranges... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2018-07-02, Query 2018-07-16
Participants:

 Description   

As part of the work to improve the $out stage, we would like to work towards a model where multiple shards can perform the "merger" work for the pipeline in parallel. Specifically, when we are outputting to a sharded collection from a sharded collection, we would like each shard to partition the output data to each other shard based on the output shard key. This is in contrast to our current sharded execution strategy, which would gather all the results from each shard to one "merger" process, which would then have to scatter the writes across the cluster.

We believe the "exchange" operator/model would be useful here, and that the AsyncResultsMerger fits pretty well as the consumer part of the exchange. We also believe that the TeeBuffer class is pretty close to acting as a producer, but needs to be remodeled and adapted to serve requests from multiple threads.



 Comments   
Comment by Githook User [ 06/Jul/18 ]

Author:

{'username': 'MartinNeupauer', 'name': 'Martin Neupauer', 'email': 'martin.neupauer@10gen.com'}

Message: SERVER-35894 The initial implementation of the producer document source
for the exchange operator.
SERVER-35940 Remove IntrusiveCounter in favor of RefCountable.
Branch: master
https://github.com/mongodb/mongo/commit/fda766f6be1a20fa28ce361511bc62e5c995186b

Generated at Thu Feb 08 04:41:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.