ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Unknown
    • 10.0.3, 9.1.3, 8.4.3
    • Affects Version/s: None
    • Component/s: None
    • None
    • Dotnet Drivers
    • Not Needed
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      The cross-collection $lookup path hard-codes preserveNullAndEmptyArrays: true on every $unwind it emits, which is left-outer-join semantics. EF Core lowers a required reference navigation to an inner Queryable.Join and an optional one to a LeftJoin, so a required navigation currently gets left-outer semantics where it should get an inner join. MongoDB has no referential integrity, so a document whose foreign key matches nothing is an ordinary data state, and such a row is currently returned with a null navigation instead of being excluded.

      Measured consequence: Where_join_orderby_join_select in the Northwind specification suite returns 2145 rows where 2143 is correct.

      This is silent wrong data. It is also what blocks a clean fix for EF-369, and what keeps four EF-216 tests skipped.

      Scope: unreleased, no released version affected

      The affected code arrived with efb5f25 ("EF-117: Cross-collection Include / navigations / joins", PR #309, 2026-06-11), which git merge-base --is-ancestor confirms is not an ancestor of v10.0.2. The three hard-coded sites do not exist in v8.4.2, v9.1.2 or v10.0.2.

      So no upgrading consumer can observe a change, and this is not a breaking change under the versioning rubric — which is precisely what makes it affordable to make the join semantics correct rather than bug-compatible. That window closes at the next release.

      The three sites

      preserveNullAndEmptyArrays: true is hard-coded in:

      • MongoEFToLinqTranslatingExpressionVisitor.LeftJoin.cs, EmitLookupStages
      • MongoEFToLinqTranslatingExpressionVisitor.LeftJoin.cs, TryBuildDriverNativeLeftJoinPipeline
      • MongoProjectionBindingExpressionVisitor.Lookup.cs, the collection-Include unwind

      Fix

      Add a PreserveNullAndEmptyArrays flag to LookupExpression, thread an isLeftOuter value from the three existing {{Translate

      {Join,LeftJoin,GroupJoin}

      }} overrides into TranslateJoinCore, and emit a non-preserving $unwind for an inner join. Prototyped at roughly 30 lines.

      The discriminator must be the LINQ operator, with ForeignKey.IsRequired only as a fallback. IsRequired alone is insufficient, and this was measured rather than assumed: Where_join_orderby_join_select joins Customers to Orders on CustomerID, which resolves to the Customer.Orders navigation, whose foreign key Order.CustomerID is nullable. An IsRequired-keyed rule therefore still emits a preserving unwind for it and still returns 2145. The operator is authoritative and is observable at the join-registration site. The one site where it is not observable is the retroactive re-registration of a prior driver-native join, and there ForeignKey.IsRequired is the correct model-derived fallback.

      Test impact: mechanical, classified, no regressions

      Prototype measurements against the specification suites: 114 case failures on EF10, 22 on EF8/EF9 (10 of the 22 an artefact of temporarily un-skipping the EF-216 tests).

      Cases Kind Verdict
      104 EF10 + 8 EF8/EF9 preserveNullAndEmptyArrays flips true to false in the Include suites Correct. An inner join is what a required navigation should emit, and what relational EF Core emits. Every one had its base.X() data assertion pass before reaching AssertMql. Pure re-baseline.
      2 + 2 Multiple_joins_Where_Order_Any MQL path change Data assertion passes. Re-baseline.
      4 + 2 Overrides that now succeed where they previously asserted a translation failure Fixes. Need rewriting to assert the correct data.
      2 Exception type changes Not contract for an unsupported shape, per the versioning rubric.
      2 Pre-existing Select_Where_Navigation_Null_Deep Unrelated, tracked separately.

      Zero regressions, no new wrong data, and no row set got worse.

      Because MQL re-baselines are data-gated by construction (AssertMql is the last call in an override, after await base.X(async)), the 112 mechanical rewrites can only be produced once the data assertions already pass — so the churn is safe to regenerate with EF_TEST_REWRITE_BASELINES=1, then rebuild and re-run.

      What this unblocks

      • EF-369. A fix gated to all-LeftJoin chains is verified green with zero churn but does not fix the required-navigation variant on any version. With the unwind semantics correct, that gate is unnecessary, and the synthesized-versus-user-authored Join distinction — genuinely undecidable, since both are Queryable.Join with a transparent-identifier selector over a bare root and both carry a real FK/PK key pair — stops mattering, because both should be inner.
      • EF-216. Four of the five skipped NorthwindNavigationsQueryMongoTest cases pass with this change, plus Select_Where_Navigation_Deep. See the comment on that ticket.
      • EF-368 (native reference Include, on the unmerged native branch) needs the same decision for its own lowerer. Making it once here and sharing it avoids two divergent answers.

      Sequencing

      This is slice 1 of two. It absorbs all the baseline churn and is independently correct, so it can ship alone. EF-369's ungated reattach is slice 2 and is then churn-free.

      Ordering constraint inherited from EF-369

      A separate finding recorded on EF-369: the bulk ExecuteUpdate/ExecuteDelete path is protected from the EF-369 defect only incidentally, by ClassifyBulkSource rejecting EF's trailing Select. Both slices should land before any widening of the bulk source classifier.

            Assignee:
            Arthur Vickers
            Reporter:
            Arthur Vickers
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: