Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-56583

Push $setWindowFields to shards when shards contain whole partitions

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Backlog
    • Component/s: None
    • Labels:
      None
    • Sprint:
      QE 2021-10-18, QE 2021-11-01

      Description

      It's safe to run the $setWindowFields stage on each shard independently as long as each partition lives on a single shard. That means the shard key must be at least as coarse as the partitionBy expression / the shard key must be constant within each partition.

      For example, given this query:

      {$setWindowFields: {
        partitionBy: {state: "$state", city: "$city"},
        ...
      }}
      

      This is safe to push down if the shard key is any of these:

      • {state: 1, city: 1}
      • {city: 1, state: 1}
      • {state: 1}
      • {city: 1}

        In all these cases, the documents that the 'partitionBy' groups together live on the same shard.

      Some examples of shard keys that wouldn't allow this:

      • {state: 1, city: 1, _id: 1}
      • {_id: 1}
      • {country: 1, state: 1, city: 1}

      The analysis we'd need seems similar to pushing down $match past $setWindowFields: SERVER-56419.

      One problem is that we convert $setWindowFields with a partitionBy to a $sort + $_internalSetWindowFields before optimization. Pushing down the $sort wouldn't be valid.

      • Maybe we should somehow delay introducing the $sort until after we've done some optimization.
      • Maybe we can do something like SERVER-53638, where we push a $lookup down through the mergesort part of a sharded sort.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              backlog-query-execution Backlog - Query Execution
              Reporter:
              david.percy David Percy
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: