[SERVER-35419] Make diskRequirement in StageConstraints in DocumentSource reflect child pipeline requirements. Created: 05/Jun/18  Updated: 29/Oct/23  Resolved: 14/Sep/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 4.1.4

Type: Bug Priority: Major - P3
Reporter: Sam Mercier Assignee: Kyle Suarez
Resolution: Fixed Votes: 0
Labels: read-only-views
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Documented
is documented by DOCS-12086 Docs for SERVER-35419: Make diskRequi... Closed
Gantt Dependency
has to be done after SERVER-34902 view definitions should *not* allow $... Closed
Related
related to SERVER-36185 Allow specifying bypassDocumentValida... Backlog
related to SERVER-37140 Investigate what other StageConstrain... Closed
Backwards Compatibility: Minor Change
Operating System: ALL
Backport Requested:
v4.0
Sprint: Query 2018-09-24
Participants:

 Description   

Currently, if a stage (e.g. `$lookup`) has a "sub" pipeline that requires it to write persistent data, the stage's diskRequirement doesn't reflect that.

Instead the parent stage should have the max diskRequirement of it's pipeline's stages under the total ordering

kNoDiskUse < kWritesTmpData < kWritesPersistentData

e.g. of the bug https://github.com/mongodb/mongo/blob/b93fe0e61bf7e8bc96da2edeb66afa1b915b0b77/src/mongo/db/pipeline/document_source_lookup.h#L100-L112

https://github.com/mongodb/mongo/blob/b93fe0e61bf7e8bc96da2edeb66afa1b915b0b77/src/mongo/db/pipeline/document_source_facet.cpp#L232-L265 



 Comments   
Comment by Githook User [ 14/Sep/18 ]

Author:

{'name': 'Kyle Suarez', 'email': 'kyle.suarez@mongodb.com', 'username': 'ksuarz'}

Message: SERVER-35419 $lookup and $facet must inherit constraints from children

By default, $lookup and $facet do not write persistent data and are
allowed in a transaction. However, both stages must inherit the
"strictest" disk use requirement of any stage in their sub-pipelines,
and can only be used in a transaction if each of those pipelines contain
only transaction-compatible stages.
Branch: master
https://github.com/mongodb/mongo/commit/7e8486ba84837a548f2f1470209eb204a42c293a

Comment by Kyle Suarez [ 24/Jul/18 ]

I'm throwing this back into Needs Triage because the bug here allows views to be saved to the catalog when they contain a $lookup with a $out pipeline, and this affects the Upgrade/Downgrade design for the Improved $out project.

Comment by David Storch [ 15/Jun/18 ]

kyle.suarez suggests that due to this bug, the view catalog will permit a $lookup into a sub-pipeline with $out.

Comment by Ian Boros [ 14/Jun/18 ]

david.storch I don't think it will impact which node is chosen as a merger:

We only know of this bug in two places: $facet and $lookup. The bug only affects stages which have a subpipeline with a stage that has DiskUseRequirement of kWritesPersistentData. The only stage with this DiskUseRequirement is $out, so there are only a few possibilities to consider.

$facet explicitly bans $out from running in a subpipeline.

$lookup has the HostTypeRequirement of kPrimaryShard, meaning that it will always be run on on a shard anyway (see this).

So while this is a "bug" I can't see how it can have any consequences right now, unless stages other than $lookup and $facet can have subpipelines .

Comment by Sam Mercier [ 13/Jun/18 ]

david.storch I ran into this issue because it impacts the solution to SERVER-34902. I'm unaware (due to lack of investigation & familiarity with the codebase) of other impacts.

Comment by David Storch [ 13/Jun/18 ]

samuel.mercier kyle.suarez, have you investigated the impact of this bug from a user perspective? Does it just impact whether a mongos node is nominated as a merger?

Generated at Thu Feb 08 04:39:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.