-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Optimization
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Currently, in the join optimizer, the ExpCtx lifetime is quite tricky and difficult to follow due to the fact that generally, the mongo query engine was designed around the assumption of a "main" collection and "secondary" collections which no longer holds with joins.
Current behavior:
1. JOO begins with a speculative copy
AggJoinModel::constructJoinModel() obtains the original pipeline context and creates clonedExpCtx with makeCopyFromExpressionContext(). It then clones the pipeline—including subpipelines—with the cloned context. The purpose is to allow JOO to mutate the cloned pipeline while keeping the user-visible pipeline and original context unchanged if join optimization later bails out.
See agg_join_model.cpp.
The source comment says that the original expCtx should not be used below except for a temporary plan-cache adjustment, and that join-optimization work should use clonedExpCtx. The base-CQ construction currently appears to pass expCtx to createCQForJoinPipeline(), however. This should be confirmed and either made intentional or corrected.
2. JOO creates multiple CQs
The join model contains one access-path CQ per join-graph node, plus an originalFilter snapshot. These CQs do not necessarily represent one common namespace or one common context:
- The base collection CQ is created from the pipeline prefix.
- A foreign $lookup CQ is created from the lookup subpipeline and uses lookup.getSubpipelineExpCtx().
- MutableJoinGraph::addNode() creates an originalFilter snapshot from the node CQ before inference changes its access path.
- Predicate inference may replace a node's access-path CQ with a new CQ containing an inferred filter.
See join_graph.cpp and agg_join_model.cpp.
3. Single table predicate inferences ExpCtx transformations
The current single-table-predicate (STP) path is:
- addImplicitEdgesAndInferPredicates() receives clonedExpCtx.
- STPs are extracted and combined using that context.
- finalizeSTPForTargetNode() uses the target node's existing CQ context to:
- clone an aggregation Expression when the STP came from $expr;
- construct the ExprMatchExpression; and
- rewrite field paths for the target node.
- Because CanonicalQuery's match expression is read-only after construction, propagation creates a replacement CQ with the new filter. The replacement CQ currently receieves a newly built context that contains only the target namespace and operation context, meaning it loses other information like collation, path arrayness information, etc (see SERVER-130378)
- The replacement CQ is installed as the node's new access path, while the original CQ is retained as the originalFilter and the model retains backing BSON needed by the match expressions.
As a result, by the time we lower to SBE, it is not clear what the ExpCtx of the "base" CQ we are passing to SBE is.
We should investigate the ExpCtx lifetimes, information loss and see if there is a way to codify it.
- is related to
-
SERVER-130378 Pass targetNode's ExpCtx info when creating CanonicalQuery in predicate inference
-
- In Code Review
-