[SERVER-56757] $setWindowFields getting nondeterministic order of input would output nondeterministic result if ‘sortBy’ a non-existing field Created: 07/May/21  Updated: 29/Oct/23  Resolved: 25/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc1, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Ruoxin Xu Assignee: Sam Mercier
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Query Optimization 2021-05-17, Query Optimization 2021-05-31
Participants:
Linked BF Score: 130

 Description   

If the preceding stages generate output in nondeterministic order and $setWindowFields has a default ‘partitionBy’ and ‘sortBy’ a non-existing field, $setWindowFields could generate inconsistent result every time the same pipeline is executed.

Due to the same reason, disabling/enabling query optimization could also cause $setWindowFields producing inconsistent result. For example in BF-20745, consider a pipeline [*$bucket + $setWindowFields]* which will be desugared to the following pipeline: [ $group + $sort ({“_id”: 1})  + $sort + $_internalSetWindowFields ]. The two adjacent $sort stages will be optimized to a single $sort by removing the first $sort. However, if the second $sort sorted a non-existing field, $_internalSetWindowFields would still get a non-deterministic order of input because $group does not order its output.

 

It seems not a legit $setWindowFields bug, but it’s failing in some certain agg-fuzzers, e.g. aggregation_wildcard_fuzzer and aggregation_optimization_fuzzer.

 



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 26/May/21 ]

Author:

{'name': 'samontea', 'email': 'merciers.merciers@gmail.com', 'username': 'samontea'}

Message: SERVER-56757 Add query knob to ensure that setWindowFields is in deterministic order for testing framework
Branch: v5.0
https://github.com/mongodb/mongo/commit/5929fa77a2393235b541f5c26747907822b5dfdd

Comment by Githook User [ 25/May/21 ]

Author:

{'name': 'samontea', 'email': 'merciers.merciers@gmail.com', 'username': 'samontea'}

Message: SERVER-56757 Add query knob to ensure that setWindowFields is in deterministic order for testing framework
Branch: master
https://github.com/mongodb/mongo/commit/9d31696a31388af10b3351ac6f67ad1765d1d48c

Comment by Sam Mercier [ 24/May/21 ]

We're going to implement a setParameter which will allow cause the `sortBy` to desugar to a `$sort` statement with and `, _id: 1` part appended. Then we will enable this setParameter with TIG-3025.  

Comment by Julian Edwards [ 12/May/21 ]

Ignore that merge, my ticket was SERVER-56575.

Comment by Githook User [ 12/May/21 ]

Author:

{'name': 'julianedwards', 'email': 'julian.edwards@mongodb.com'}

Message: SERVER-56757: Make jasper resmoke work with windows
Branch: master
https://github.com/mongodb/mongo/commit/b98b00f9d3d15f229762a9e2491a555fc9a7f93c

Comment by Nicholas Zolnierz [ 10/May/21 ]

Assigning to Sam as more of an investigating ticket. Our last brainstorm idea was to have the fuzzer append "_id" to the generated sort spec for $setWindowFields, such that the order of documents coming into the stage is reliable since the fuzzer appears to preserve the _id field. One caveat is that we'll need to perform the desugaring in the fuzzer since a compound sortBy is not allowed with several window functions.

Generated at Thu Feb 08 05:40:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.