[SERVER-36787] An $out is not eligible for $exchange if a $project attempts a "rename" with a dotted field Created: 21/Aug/18  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Optimization
Participants:

 Description   

Suppose your run the following aggregation:

db.test.aggregate([
  {$group: {
    _id: {
      date: {$dateToString: {date: "$_id", format: "%Y-%m-%d"}},
      region: "$region"
    },
    total: {$sum: 1}
  }},
  {$project: {_id: 0, region: "$_id.region", date: "$_id.date", total: 1}},
  {$out: {to: "sharded_by_region", mode: "replaceDocuments", uniqueKey: {region: 1, _id: 1}}}
])

Further suppose that the collection "sharded_by_region" has shard key {region: 1}. It looks like this pipeline is eligible for an $exchange optimization because all the way from the $group to the $out the shard key is preserved - it's just renamed from "_id.region" to top-level "region".

Unfortunately, our dependency/rename tracking will not consider this to be a strict rename, because it cannot figure out that "_id" won't be an array. If "_id" were an array, than the $project stage would be doing more than a rename, instead transforming the array previously stored in "_id" and storing the result of the transformation in "region" or "date" accordingly.

This use-case of using a $group with multiple group-by keys seems common enough for us to consider adding custom logic to communicate to the dependency/rename tracking system that we know that either (1) "_id" is not an array or (2) the pipeline will result in an error because the shard key and the uniqueKey cannot contain arrays.


Generated at Thu Feb 08 04:44:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.