Native LINQ query provider (ground-up rebuild)

XMLWordPrintableJSON

    • Type: Epic
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Native LINQ query provider (ground-up rebuild)
    • Dotnet Drivers
    • Not Needed
    • Transparent to users
    • In Progress
    • None
    • 0
    • 0
    • 0
    • None
    • None
    • Hide

      2026-07-29 - No confidence status provided
      No project update provided


      Show
      2026-07-29 - No confidence status provided No project update provided
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Ground-up native LINQ query provider for the MongoDB EF Core provider.

      What we are doing

      Replace the translation half of the Query subsystem. The provider will build MongoDB aggregation pipelines (MQL) itself from a canonical query AST and use the C# driver only to execute them (BSON, cursors, sessions, transactions) — the driver's LINQ provider is no longer the translation engine. Keep the streaming materializer and the test/benchmark rig; delete the driver-LINQ delegation once native reaches parity.

      Architecture: a Mongo query AST → typed stage IR → BsonDocument[] pipeline, built once at compile time and parameter-bound per execution. Pipeline choice is a user-level DbContext config option (Native default / DriverLinq revert / NativeStrict diagnostic) — no environment variables, so a user who hits a regression can switch back to the legacy pipeline. Materialization endgame: read stream → POCO in one pass, with no intermediate BSON objects and no double copy.

      Why

      Performance (N=10k):

      Shape Current provider Native
      Where→ToList 15.1 ms / 22.7 MB 8.3 ms / 9.6 MB
      Whole-entity (no-track) 33.4 ms / 45.3 MB 15.6 ms / 19.1 MB
      Reference Include 138.3 ms / 52.0 MB 114.7 ms / 22.3 MB

      ~50% less allocation and ~45–53% faster on allocation-heavy reads.

      Conformance ceiling. The driver's LINQ provider lacks operators and was not built to EF Core semantics, so a class of spec tests cannot pass while we delegate to it — and cannot be fixed without owning translation. Owning it lowers the limit to "what MongoDB can express," the only limit worth having.

      Delivery (sub-projects, each at zero regressions with driver-LINQ as the fallback)

      1. AST foundation — filter / sort / paging + single-level reference Include. (designed, ready to plan)
      2. Predicate breadth — the $expr renderer and the operator long tail.
      3. Projection pushdown — server-side $project.
      4. Scalar cardinalityCount / First / Any / aggregates.
      5. Collection Includes — collection / nested / filtered.
      6. Remaining operatorsGroupBy, SelectMany, set ops, Distinct, OfType, VectorSearch, non-canonical paging.
      7. Materializer perf — one-pass stream → POCO (no per-row BSON object).

      At parity: retire the driver-LINQ fallback and delete the delegation code.

      References

      Design docs (on the feature branch, off main):

      • docs/superpowers/specs/2026-06-23-native-query-provider-overview.md — terse overview / review entry point
      • docs/superpowers/specs/2026-06-23-native-query-provider-design.md — full design
      • docs/superpowers/specs/2026-06-20-mongo-query-ast-foundation-design.md — sub-project 1 detail

      Spike reference (proof-of-concept, not merged): branch spike/low-level-provider.

      Success metric: EF Core spec conformance. Per sub-project: zero regressions, native coverage must not shrink, benchmarks ≥ spike.

            Assignee:
            Arthur Vickers
            Reporter:
            Arthur Vickers
            None
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              11 weeks
              None
              None
              None