[SERVER-56757] $setWindowFields getting nondeterministic order of input would output nondeterministic result if ‘sortBy’ a non-existing field Created: 07/May/21 Updated: 29/Oct/23 Resolved: 25/May/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.0-rc1, 5.1.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Ruoxin Xu | Assignee: | Sam Mercier |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v5.0
|
||||||||
| Sprint: | Query Optimization 2021-05-17, Query Optimization 2021-05-31 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 130 | ||||||||
| Description |
|
If the preceding stages generate output in nondeterministic order and $setWindowFields has a default ‘partitionBy’ and ‘sortBy’ a non-existing field, $setWindowFields could generate inconsistent result every time the same pipeline is executed. Due to the same reason, disabling/enabling query optimization could also cause $setWindowFields producing inconsistent result. For example in BF-20745, consider a pipeline [*$bucket + $setWindowFields]* which will be desugared to the following pipeline: [ $group + $sort ({“_id”: 1}) + $sort + $_internalSetWindowFields ]. The two adjacent $sort stages will be optimized to a single $sort by removing the first $sort. However, if the second $sort sorted a non-existing field, $_internalSetWindowFields would still get a non-deterministic order of input because $group does not order its output.
It seems not a legit $setWindowFields bug, but it’s failing in some certain agg-fuzzers, e.g. aggregation_wildcard_fuzzer and aggregation_optimization_fuzzer.
|
| Comments |
| Comment by Vivian Ge (Inactive) [ 06/Oct/21 ] |
|
Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you! |
| Comment by Githook User [ 26/May/21 ] |
|
Author: {'name': 'samontea', 'email': 'merciers.merciers@gmail.com', 'username': 'samontea'}Message: |
| Comment by Githook User [ 25/May/21 ] |
|
Author: {'name': 'samontea', 'email': 'merciers.merciers@gmail.com', 'username': 'samontea'}Message: |
| Comment by Sam Mercier [ 24/May/21 ] |
|
We're going to implement a setParameter which will allow cause the `sortBy` to desugar to a `$sort` statement with and `, _id: 1` part appended. Then we will enable this setParameter with TIG-3025. |
| Comment by Julian Edwards [ 12/May/21 ] |
|
Ignore that merge, my ticket was |
| Comment by Githook User [ 12/May/21 ] |
|
Author: {'name': 'julianedwards', 'email': 'julian.edwards@mongodb.com'}Message: |
| Comment by Nicholas Zolnierz [ 10/May/21 ] |
|
Assigning to Sam as more of an investigating ticket. Our last brainstorm idea was to have the fuzzer append "_id" to the generated sort spec for $setWindowFields, such that the order of documents coming into the stage is reliable since the fuzzer appears to preserve the _id field. One caveat is that we'll need to perform the desugaring in the fuzzer since a compound sortBy is not allowed with several window functions. |