[SERVER-36364] For sharded $out, select merging shard only from set of shards that might accept writes Created: 30/Jul/18  Updated: 06/Dec/22  Resolved: 03/Aug/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Kyle Suarez Assignee: Backlog - Query Team (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-35904 Implement logic to detect if merging ... Closed
Assigned Teams:
Query
Participants:

 Description   

Consider a sharded $out aggregation whose input documents come from, say, shard0 and shard1, but sends it output to a different sharded collection whose chunks reside on shard2, shard3 and shard4. Currently, only shard0 and shard1 are eligible for selection as the merging shards. Instead, we will plumb through information during the cluster aggregation planning process such that the planner knows to select one of {shard2, shard3, shard4} as the merging shard, in the hopes that it makes more of those writes local.

However, it's out of scope for this ticket to take chunk migrations into account. For example, if a chunk migrates from shard4 to shard2 in the middle of aggregation such that shard4 no longer owns chunks for the output collection, we will still continue with an aggregation plan that chose to merge on shard4.



 Comments   
Comment by Kyle Suarez [ 01/Aug/18 ]

After an in-person discussion, we've agreed with asya that this is out-of-scope for the Improved $out project. However, we could still do this in the future – let's decide at Needs Triage whether to throw it onto the backlog or close it as Won't Fix.

Comment by Charlie Swanson [ 30/Jul/18 ]

kyle.suarez this description looks correct.

One consideration: when there's a large amount of data involved, but the merging pipeline is responsible for cutting down the cardinality of documents (e.g. a late $match), it might have been better to merge on one of the producer shards before sending data to the shards that might accept writes. Perhaps we should mention that the "Improved $out" project might benefit from the "Better explain / Direct execution" project?

Do you mean that this would be faster because the non-matching data wouldn't have to be shipped across the network before being discarded? I buy that it might be slower. I'm personally not all that concerned with this case because we have a similar problem today where one participating shard may not contribute much to the aggregation but be chosen as the merger. It would theoretically be more optimal to chose the merger based on which shard has the most results to merge - which of course we couldn't detect very easily.

Comment by Kyle Suarez [ 30/Jul/18 ]

asya, charlie.swanson: is this description accurate?

One consideration: when there's a large amount of data involved, but the merging pipeline is responsible for cutting down the cardinality of documents (e.g. a late $match), it might have been better to merge on one of the producer shards before sending data to the shards that might accept writes. Perhaps we should mention that the "Improved $out" project might benefit from the "Better explain / Direct execution" project?

Generated at Thu Feb 08 04:42:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.