Improve explain CE for plans containing $limit

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Optimization
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      $limit affects the CE of stages both before and after it in a pipeline.

      Currently propagateLimit changes the CE source to Metadata. This is reasonable for the limit itself, but when propagated to children this can be misleading.

      For example, for a plan

      LIMIT <- IXSCAN
      

      Under sampling CE, the IXSCAN would have CE source Sampling, but if the limit caps the IXSCAN CE, this gets overwritten to Metadata.

      However, neither Metadata or Sampling alone conveys the full truth.

      • Sampling but with a limited CE -> Confusion as to why it is a much lower CE than expected
      • Metadata (or a new potential value like Limited) -> Unclear what the original source before limiting was (Sampling? Histograms?)

      Additionally, the original, unlimited CE may be useful for debugging.

      Given this, retaining the previous CE (including source) should be investigated.

      Limit is the only stage expected to affect children like this. However, the solution to this could be future-proof by expressing a list of "intermediate" CEs computed during cardinality estimation.


      Evaluate possible ways to express this in explain for a given stage. For example,

      // Current
      "stage": "IXSCAN",
      "cardinalityEstimate": 100,
      "estimateMetadata": {"ceSource": "Limit"},
      // Potential addition
      "estimateHistory": [{"ce":1432, "ceSource": "Sampling"}],
      

      To be clear, this is for human/test/debugging consumption; only the final CE is relevant for subsequent costing.

            Assignee:
            Unassigned
            Reporter:
            James Harrison
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: