Include specification suites mask wrong-data failures behind a bare-catch AssertTranslationFailed shadow

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Unknown
    • 10.0.4, 9.1.4, 8.4.4
    • Affects Version/s: None
    • Component/s: None
    • None
    • Dotnet Drivers
    • Not Needed
    • None
    • None
    • None
    • None
    • None
    • None

      The defect

      Four specification-test suites each declare their own shadow of AssertTranslationFailed:

      Unable to find source-code formatter for language: csharp. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      protected new static async Task AssertTranslationFailed(Func<Task> query)
      {
          try { await query(); }
          catch { return; }          // treats WRONG DATA as success
          throw new XunitException("Expected query to fail but it succeeded.");
      }
      

      The bare catch swallows the base test's result-mismatch assertion, which is just another exception. So a query that executes and returns the wrong rows counts as a pass. The only thing that can fail is a query that succeeds and returns something.

      Affected suites (all under tests/MongoDB.EntityFrameworkCore.SpecificationTests/Query/):

      1. NorthwindIncludeQueryMongoTest
      2. NorthwindIncludeNoTrackingQueryMongoTest
      3. NorthwindStringIncludeQueryMongoTest
      4. NorthwindEFPropertyIncludeQueryMongoTest

      Between them there are 234 call sites of the shadowed helper.

      The other spec suites do not have this problem: they route through MongoSpecTestHelpers.AssertNativeTranslationFailedAsync, which accepts only a known set of translation-failure exception types and rethrows xUnit assertion failures, so wrong data fails there.

      Why it matters

      This is not hypothetical. The strict/masking split is the only reason the driver 3.10 silent-wrong-data regression fixed under EF-366 was caught at all — it surfaced in NorthwindGroupByQueryMongoTest, which uses the strict helper. An identical wrong-data regression in any of the four Include suites would have been swallowed and the suite would have stayed green.

      Those four suites are also exactly the suites the planned native reference-Include work would use to verify itself, so they need to be trustworthy before that work lands.

      Expected outcome

      Make the four suites fail on wrong data. Fixing the helper is expected to reveal a number of pre-existing failures; those are defects being surfaced, not caused, and each needs triaging on its merits — some will be genuine provider bugs to baseline honestly per the project convention, some will be tests whose "translation fails" premise is no longer true.

      Measurement of exactly how many of the 234 call sites currently mask a real failure, and what each masked failure is, is the first task.

            Assignee:
            Arthur Vickers
            Reporter:
            Arthur Vickers
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: