[SERVER-56583] Push $setWindowFields to shards when shards contain whole partitions Created: 03/May/21  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: David Percy Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-53638 Enable pushdown of config.cache.chunk... Closed
is related to SERVER-5477 when sharded, no need to merge groups... Backlog
is related to SERVER-56419 Push down $match past $setWindowField... Backlog
Assigned Teams:
Query Execution
Sprint: QE 2021-10-18, QE 2021-11-01, QE 2021-11-15, QE 2021-11-29, QE 2021-12-13, QE 2021-12-27, QE 2022-01-10, QE 2022-02-07, QE 2022-02-21, QE 2022-01-24
Participants:

 Description   

It's safe to run the $setWindowFields stage on each shard independently as long as each partition lives on a single shard. That means the shard key must be at least as coarse as the partitionBy expression / the shard key must be constant within each partition.

For example, given this query:

{$setWindowFields: {
  partitionBy: {state: "$state", city: "$city"},
  ...
}}

This is safe to push down if the shard key is any of these:

  • {state: 1, city: 1}
  • {city: 1, state: 1}
  • {state: 1}
  • {city: 1}

    In all these cases, the documents that the 'partitionBy' groups together live on the same shard.

Some examples of shard keys that wouldn't allow this:

  • {state: 1, city: 1, _id: 1}
  • {_id: 1}
  • {country: 1, state: 1, city: 1}

The analysis we'd need seems similar to pushing down $match past $setWindowFields: SERVER-56419.

One problem is that we convert $setWindowFields with a partitionBy to a $sort + $_internalSetWindowFields before optimization. Pushing down the $sort wouldn't be valid.

  • Maybe we should somehow delay introducing the $sort until after we've done some optimization.
  • Maybe we can do something like SERVER-53638, where we push a $lookup down through the mergesort part of a sharded sort.


 Comments   
Comment by Colby Ing [ 04/Nov/22 ]

Probably not high urgency, as 1% of customers use sharding and $setWindowFields is used by even fewer customers. Unless there's a specific customer who complains, I would say this is relatively low priority

Comment by Kyle Suarez [ 02/Mar/22 ]

This ticket got lost on the backlog; flagging for retriage.

Generated at Thu Feb 08 05:39:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.