Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-56583

Push $setWindowFields to shards when shards contain whole partitions

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • QE 2021-10-18, QE 2021-11-01, QE 2021-11-15, QE 2021-11-29, QE 2021-12-13, QE 2021-12-27, QE 2022-01-10, QE 2022-02-07, QE 2022-02-21, QE 2022-01-24

      It's safe to run the $setWindowFields stage on each shard independently as long as each partition lives on a single shard. That means the shard key must be at least as coarse as the partitionBy expression / the shard key must be constant within each partition.

      For example, given this query:

      {$setWindowFields: {
        partitionBy: {state: "$state", city: "$city"},
        ...
      }}
      

      This is safe to push down if the shard key is any of these:

      • {state: 1, city: 1}
      • {city: 1, state: 1}
      • {state: 1}
      • {city: 1}

        In all these cases, the documents that the 'partitionBy' groups together live on the same shard.

      Some examples of shard keys that wouldn't allow this:

      • {state: 1, city: 1, _id: 1}
      • {_id: 1}
      • {country: 1, state: 1, city: 1}

      The analysis we'd need seems similar to pushing down $match past $setWindowFields: SERVER-56419.

      One problem is that we convert $setWindowFields with a partitionBy to a $sort + $_internalSetWindowFields before optimization. Pushing down the $sort wouldn't be valid.

      • Maybe we should somehow delay introducing the $sort until after we've done some optimization.
      • Maybe we can do something like SERVER-53638, where we push a $lookup down through the mergesort part of a sharded sort.

            Assignee:
            backlog-query-execution [DO NOT USE] Backlog - Query Execution
            Reporter:
            david.percy@mongodb.com David Percy
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: