[SERVER-17737] Support distributed merger for aggregation queries Created: 25/Mar/15  Updated: 06/Dec/22

Status: Open
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Jon Rangel (Inactive) Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 6
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documented
is documented by DOCS-7237 Ambiguity about Where the Merge Sort ... Closed
is documented by DOCS-11469 Ambiguity about Where the Merge Sort ... Closed
Duplicate
is duplicated by SERVER-14985 Merge stages in aggregation should be... Closed
Related
is related to SERVER-18925 Merging part of aggregation pipeline ... Closed
Assigned Teams:
Query Optimization
Participants:
Case:

 Description   

As described in the MongoDB Manual (http://docs.mongodb.org/manual/core/aggregation-pipeline-sharded-collections/), from 2.6 onwards aggregation pipelines are executed as follows in a sharded cluster:

When operating on a sharded collection, the aggregation pipeline is split into two parts. The first pipeline runs on each shard, or if an early $match can exclude shards through the use of the shard key in the predicate, the pipeline runs on only the relevant shards.

The second pipeline consists of the remaining pipeline stages and runs on the primary shard. The primary shard merges the cursors from the other shards and runs the second pipeline on these results. The primary shard forwards the final results to the mongos.

For long-running aggregation queries that aggregate a lot of data, the second part of the pipeline (running on the primary shard) is a bottleneck to performance where access to data is sufficiently fast that the query becomes bound by CPU, not I/O. The second part of the pipeline runs in a single thread on the primary shard.

To improve performance for such long-running CPU-bound aggregations, it would be good to add multiple levels of merging such that the 'merger' role can be distributed to multiple shards. For example, in a sharded cluster with 16 shards, have 4 'first level' mergers each of which are responsible for merging the results from 4 shards, then a 'second level' merger which merges the results from the first level mergers and returns the result to mongos.



 Comments   
Comment by Kaloian Manassiev [ 15/Nov/21 ]

This doesn't seem like a Sharding feature, but more of Query-related infrastructure.

Comment by Andy Schwerin [ 26/Mar/15 ]

Good idea, a lot of work.

Generated at Thu Feb 08 03:45:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.