Make $_internalValidateArrayness transparent to SBE pushdown

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Optimization
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The $_internalValidateArrayness stage (inserted by the dependency graph validation passthrough suite) blocks SBE pushdown of subsequent stages, causing tests that rely on SBE plan caching/replanning to fail. Specifically, sbe_pushdown_replan_reason.js fails its $lookup test case because the validation stage inserted between $match and $lookup prevents $lookup from being pushed down to SBE.

      Background

      When internalEnableDependencyGraphValidation: true is set, the pipeline optimizer inserts $_internalValidateArrayness stages before each pipeline stage to validate the dependency graph's canPathBeArray() analysis at runtime. The optimized pipeline for the failing test becomes:

      The SBE pushdown logic in pipelineStageIsCompatible() (sbe_pushdown.cpp) iterates stages sequentially and stops at the first unrecognized stage. Since $_internalValidateArrayness is not in the recognized stage list, it falls through to return false, breaking the pushdown chain and preventing $lookup (and any stage after the validation stage) from being pushed to SBE.

      Impact

      sbe_pushdown_replan_reason.js (lookup variant) must be excluded from the aggregation_dependency_graph_validation_passthrough suite
      Any test relying on multi-stage SBE pushdown will fail or need exclusion when dependency graph validation is enabled
      Reduces test coverage of the dependency graph validation suite

      Proposed Solution

      Make $_internalValidateArrayness SBE-compatible by treating it as a transparent passthrough in the pushdown logic. Since this stage only validates documents without modifying them (it's a streaming, no-op passthrough), it is safe to push it down alongside other stages. Implementation steps:

      Add a validateArrayness field to CompatiblePipelineStages in sbe_pushdown.cpp
      Add an else if (stageId == DocumentSourceInternalValidateArrayness::id) branch in pipelineStageIsCompatible() that unconditionally returns true (the stage has no expressions that could be SBE-incompatible)
      Set the flag in findSbeCompatibleStagesForPushdown() — gated on SbeCompatibility::noRequirements since this is test-only infrastructure
      Implement an SBE execution node for the stage (or have SBE skip it entirely since it's a passthrough that only needs to validate documents)
      Alternatively, a simpler approach: modify the validation stage injection logic to skip insertion when the next stage would be pushed down to SBE, or have the pushdown logic skip over $_internalValidateArrayness stages transparently (treating them as invisible). This avoids needing a full SBE execution node.

            Assignee:
            Unassigned
            Reporter:
            Matt Olma
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: