[SERVER-22760] Sharded aggregation pipelines which involve taking a simple union should merge on mongos Created: 19/Feb/16 Updated: 20/May/21 Resolved: 08/Aug/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.5.12 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | David Storch | Assignee: | Bernard Gorman |
| Resolution: | Done | Votes: | 0 |
| Labels: | eng-l, optimization | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Query 13 (04/22/16), Query 2017-07-10, Query 2017-07-31, Query 2017-08-21 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Suppose you have a pipeline consisting of a single $match stage which is delivered to mongos by the client. Also suppose that mongos will target multiple shards in order to find the matching data. Currently, the aggregation system will establish a shard as the merging node. Instead, mongos could perform the merging (it already knows how to do this for .find() operations). This would improve the latency of the aggregation operation, since it would eliminate a network hop. Data currently has to travel from a targeted shard, to the merging shard, to mongos, and back to the client. If mongos were responsible for merging, data would only have to travel from a targeted shard, to mongos, to the client. Mongos can also merge sorted streams coming from the shards for .find() operations, so this could also be extended to $match+$sort pipelines. |
| Comments |
| Comment by Githook User [ 08/Aug/17 ] | |||||||||||||||||||||||||||||||||||
|
Author: {'username': 'gormanb', 'email': 'bernard.gorman@gmail.com', 'name': 'Bernard Gorman'}Message: | |||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 11/Mar/16 ] | |||||||||||||||||||||||||||||||||||
That's okay, IMHO - I think biggest win would be if find ( expression ) and aggregate match same-expression have the same performance in sharded cluster. | |||||||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 09/Mar/16 ] | |||||||||||||||||||||||||||||||||||
|
I agree with Charlie's preferred path. I don't think we should forgo optimizing agg latencies just because sometimes we cannot optimize them. The big reason in my mind when we moved the later pipeline stages to a merge mongod was to enable spilling to disk, but for queries that do not need it, and we didn't want to spend the time back then to support running the end of the pipe in either mongod or mongos. I think as Mathias says, this was in large part because it was hard to make cursors on mongos at the time. If we build a more intelligent agg planner, that detects when mongos can safely act as the merger, we'll maximize the utility of the system in the long run. | |||||||||||||||||||||||||||||||||||
| Comment by Mathias Stearn [ 09/Mar/16 ] | |||||||||||||||||||||||||||||||||||
|
charlie.swanson, the issue with merging on mongos wasn't exactly that it used too much memory, it's that we wanted to support features that mongos couldn't either at the time (returning cursors), or potentially ever ($out, allowDiskUsage). Since we wanted to use cursors for all aggregations in new drivers, it didn't make sense to have mongos have a more optimal path if it didn't support cursors. Mongos' cursor implementation has been rewritten in 3.2 which makes it easy to support agg cursors now, so that argument disappears. There was some concern previously that agg on mongos used more resources than other operations that a mongos performs, but it seems like people are more concerned about lowering the latency of operations than getting work out of mongos. However, we should discuss with the Product and Consulting teams before making another shift here. There is an obvious trivial case of "Simple merge on mongos" that could be implemented first even if we decide not to implement the full change right away. If after splitting, the merge pipeline is empty (meaning all agg stages are performed on shards), we can just use a simple forwarding/unioning cursor in mongos. This will be helpful for users who use agg by default even for operations that could be done with find. This lays a simple groundwork that can be added to incrementally (eg just support $limit in merger) until the full "Simple merge on mongos" is implemented. Also, there is a subtle downside to "Smarter non-exact match targeting". It can cause the same query to have very different execution based on how chunks are distributed. This could be a problem for users if a "hot" query suddenly transitions to using a merger following a migration. This is somewhat mitigated by the fact that the worst case in that world is exactly what we do now, but it still reduces the predictability of performance. Currently all execution choices in sharded agg are based solely on the "shape" of the pipeline, and not on the actual values or distribution of chunks across shards. | |||||||||||||||||||||||||||||||||||
| Comment by Charlie Swanson [ 09/Mar/16 ] | |||||||||||||||||||||||||||||||||||
|
I just had a conversation with david.storch about some possible improvements with aggregation in a sharded cluster. There's this one, the linked
I think the ideal scenario would be if we implemented both "Merging and running agg stages on mongos" and "Smarter non-exact match targeting". It's my understanding that we moved away from doing merging in the mongos because stages like $group and $sort were taking up too much memory. Were there any other downsides to doing the merging on the mongos? |