[SERVER-74808] Invariant failure - isVersionInitialized() Created: 09/Mar/23  Updated: 29/Oct/23  Resolved: 11/Apr/23

Status: Closed
Project: Core Server
Component/s: Field Level Encryption, Queryable Encryption
Affects Version/s: None
Fix Version/s: 6.0.6

Type: Bug Priority: Major - P3
Reporter: Eric Rosenquist Assignee: Shreyas Kalyan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-71003 Enable featureFlagFLE2Range in mongo_... Closed
Assigned Teams:
Server Security
Backwards Compatibility: Fully Compatible
Sprint: Security 2023-04-03, Security 2023-04-17
Participants:

 Description   

I've got a C# .NET application (driver 2.19.0) and I'm querying a DB using IAggregateFluent with a match, a lookup, and a projection. This is the pretty-print of the query - it's nothing terribly fancy:

[
  {
    "$match": {
      "Timestamp": {
        "$gte": "ISODate(2023-02-20T14:40:00Z)"
      }
    }
  },
  {
    "$lookup": {
      "from": "RemoteCommandResults",
      "localField": "_id",
      "foreignField": "CommandId",
      "as": "Results"
    }
  },
  {
    "$project": {
      "_id": "$_id",
      "Command": "$Command",
      "CommandType": "$CommandType",
      "Timestamp": "$Timestamp",
      "ClientIpAddress": "$ClientIpAddress",
      "Location": "$Location",
      "SubmittedByUser": "$SubmittedByUser",
      "CompletedEndpoints": {
        "$map": {
          "input": {
            "$filter": {
              "input": "$Results",
              "as": "r",
              "cond": {
                "$ne": [
                  "$$r.WhenCompleted",
                  null
                ]
              }
            }
          },
          "as": "r",
          "in": "$$r.EndpointId"
        }
      },
      "PendingEndpoints": {
        "$map": {
          "input": {
            "$filter": {
              "input": "$Results",
              "as": "r",
              "cond": {
                "$not": {
                  "$ne": [
                    "$$r.WhenCompleted",
                    null
                  ]
                }
              }
            }
          },
          "as": "r",
          "in": "$$r.EndpointId"
        }
      }
    }
  }
] 

I'm using CSFLE with mongo_crypt_v1.dll and running on Windows.

If I don't enable encryption, i.e., leave the AutoEncryptionOptions out of the MongoClientSettings, everything works as expected.

If encryption is enabled however, either libmongocrypt.dll or mongo_crypt_v1.dll prints an assertion failure to the console window and aborts the application:

{"t":{"$date":"2023-03-09T21:04:56.738Z"},"s":"F",  "c":"ASSERT",   "id":23079,   "ctx":"thread13","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"Z:\\data\\mci\\badd693c2380d3476d7367abd345c3b9\\src\\src\\mongo/db/server_options.h","line":156}}
{"t":{"$date":"2023-03-09T21:04:56.738Z"},"s":"F",  "c":"ASSERT",   "id":23080,   "ctx":"thread13","msg":"\n\n***aborting after invariant() failure\n\n"} 

The query seems to be well formed, as evidenced by the fact that it works when the encryption libraries aren't in use, so I suspect there may be a bug in the encryption libraries and how they handle aggregation pipelines.



 Comments   
Comment by Kevin Albertson [ 13/Mar/23 ]

I got the version of mongo_crypt_v1 with this script: python markup.py --version --libpath <path to library>

I reproduced the invariant failure with version mongo_crypt_v1-dev-6.0.4. On a newer version mongo_crypt_v1-dev-6.3.0-rc0, I get an expected error: Pipeline over an encrypted collection cannot reference additional collections. [Error 2, code 51204]

Here is a test with instructions to reproduce using the Go driver: https://github.com/mongodb/mongo-go-driver/compare/master...kevinAlbs:mongo-go-driver:M556?expand=1#diff-35d11cdcb894289a5379a0fb50235f1933e872f4f591cfd2583245f43f4ee8fcR614-R748.

The invariant does not occur when the $lookup stage is removed.

I am moving this to the SERVER project since this appears isolated to the mongo_crypt_v1 shared library.

 

Question for server team:
I could not find what change fixed the server behavior. Is this something that needs to be backported to 6.0?

Comment by Eric Rosenquist [ 13/Mar/23 ]

I'm running the latest mongo_crypt_v1 for Windows. The DLL doesn't have a version stamp, but the ZIP download is named mongo_crypt_shared_v1-windows-x86_64-enterprise-6.0.4.zip and contains mongo_crypt_v1.dll dated 2023-01-17 with size 18046464 bytes.

I missed that limitation on $lookup, and I am indeed referencing a different collection! I'll stick with my current workaround of issuing extra queries to get the data from the other collection.

Thanks for looking into this.

Comment by Kevin Albertson [ 13/Mar/23 ]

The invariant failure looks unexpected. I encountered the invariant failure when using an unreleased build of mongo_crypt_v1. It was fixed with the change of SERVER-71003. I suggest trying to upgrade mongo_crypt_v1 to the latest release. Downloads are available in the Crypt Shared links here: https://www.mongodb.com/download-center/enterprise/releases

Secondly, the aggregation pipeline may be expected to error during automatic encryption if doing $lookup on a different collection. This limitation is documented here. I attempted to run that aggregation with the latest build of mongo_crypt_v1 and got this error:

csfle "analyze_query" failed: Pipeline over an encrypted collection cannot reference additional collections. [Error 2, code 51204]

Generated at Thu Feb 08 06:28:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.