Harden updateLookup and $_internalSearchIdLookup enrichment testing with per-engine lookup metrics

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • 9.1.0-rc0, 9.1.0-rc1022
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • v9.0
    • QE 2026-09-15
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      Change-stream updateLookup and $_internalSearchIdLookup share the
      SingleDocumentLookupExecutor / BatchedEnrichmentStage enrichment abstraction. This work extends
      the per-engine lookup metrics to $_internalSearchIdLookup, adds per-consumer enrich-batch
      counters, and broadens test coverage across the dimensions the shared module serves: _id shapes
      (including fallback-handled declines), clustered collections, collation, sharding and placement
      changes, concurrent DDL, batching knobs, and interrupts.

      Observability

      • Per-engine outcome and latency metrics for search idLookup under search.idLookup.sbe and
        search.idLookup.aggregation (found, notFound, notHandled, latencyMicros),
        wired in buildIdLookupExecutor().
      • Per-consumer enrich-window counters changeStreams.updateLookup.enrichBatchesStarted and
        search.idLookup.enrichBatchesStarted, making batching knobs directly verifiable.

      Test coverage

      $_internalSearchIdLookup

      • Coverage reworked around a shared mock-mongot util: _id shape matrix, unsupported shapes
        (regex/array) asserted to resolve cleanly through the fallback executor, mixed-shape batches
        with per-engine attribution, view-pipeline fallback, DDL between getMores, killOp mid-batch.
      • Batching-knob invariance verified via the new enrich-batch counters.
      • Sharding/DDL concurrency: shard-key strategy matrix, migration orphan filtering,
        parked-cursor-across-migration, view-backed orphan case.

      Change-stream updateLookup

      • Engine-attribution metrics tests across clustered/collation matrices, including mixed
        scalar/compound _id batches.
      • Batching-knob verification with engine-based eligibility.
      • DDL timing: update_lookup_before_ddl.js rework plus a new noPassthrough mid-batch file using
        the hangBeforeChangeStreamUpdateLookup fail point (mid-batch cached-acquisition staleness,
        killOp mid-batch).
      • TooManyMatchingDocuments regression test for the sharded scatter-gather case.
      • New shared helpers: expectedUpdateLookupEngine(), readUpdateLookupDelta(),
        assertChangeStreamEventsEqUnordered() / assertChangeStreamEventsEqWithDeploymentAwareness().

      Unit tests

      • SbeSingleDocumentLookupExecutor: decline paths for unsupported _id shapes (non-scalar _id
        support tracked as SERVER-134080), cached-plan survival across mid-sequence declines,
        accept-side allow-list, shard filtering, plan invalidation, lifecycle.
      • Stats recorders: all factories plus cross-consumer cell independence, enrich-batch counters,
        and throw-integrity for the aggregation executor.

      Observations recorded for follow-up

      • The view-backed local-read executor appears not to request shard filtering (orphan returned);
        pinned as a KNOWN ISSUE assertion in the sharded DDL-concurrency test.
      • The shard filter appears stale across a parked cursor even under SBE; pinned the same way.

      Related

            Assignee:
            Denis Grebennicov
            Reporter:
            Denis Grebennicov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: