• Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Optimization
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This may need to be split into multiple tickets (or SPM).

      The goal of this ticket is to allow information from the DependencyGraph to be used in the QueryPlanner. The suggested approach here is to use the DependencyGraph constructed on the original pipeline before CQ is created.

      Background
      The dependency graph could provide metadata to downstream planning and code generation after a CanonicalQuery has been created. For example, QueryPlanner::extendWithAggPipeline() could query graph-derived facts such as canPathBeArray() for a pipeline stage, instead of requiring each future consumer to extract and store every potentially useful type or path annotation in advance.

      This depends on a safety assumption that has not been established: that the graph remains valid when pipeline stages have been erased during CQ creation, provided the relevant DocumentSource is still alive. The graph’s behavior and lifetime in these cases are not covered by targeted tests or an explicit API contract.

      It also depends on the assumption that we are using the original pipeline DocumentSource, not a clone(), so that the set of stages the graph was constructed with is the same (by pointer identity).

      What we need to do

      We should test these assumptions.

      And formalise a guarantee on the graph - that as long as the DependencyGraph doesn't have any non-const methods called on it (resize(), recompute_forTest()). Also that the const methods like canPathBeArray never access the DocumentSourceContainer that might have changed (or we need to start copying it). The graph keeps intrusive_ptr to the stages that were, so it could still consult them.

      Once that is verified, we know that even if the graph is constructed on [$match, $set, $group, ...] originally, and the pipeline prefix is moved into CQ, such that we are only left with [$group, ...], as long as we keep the graph as const, we could still call graph.canPathBeArray($group, "path") with the original $group DocumentSourceGroup* and the result will be the same as it was when the pipeline was [$match, $set, $group, ...]. So the $group will see the fields from $set as computed, renamed etc.

      We could then store the graph from rule-based rewrites somewhere accessible, reachable from code such as extendWithAggPipeline. This would allow extendWithAggPipeline and other planner code to use the deps information from the original pipeline at a later point. 

      It's important for it to be after optimizePipeline, since if we construct the graph first on the original pipeline, then apply rewrites, the reordered stages can now see incorrect results, such as if there was a computed field that is now no longer compute but a collection field of the same name at that position.

            Assignee:
            Unassigned
            Reporter:
            Vesko Karaganev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: