[SERVER-70134] Upsert fails if exactly one tag exists for the QE encrypted value in the query Created: 30/Sep/22  Updated: 29/Oct/23  Resolved: 01/Dec/22

Status: Closed
Project: Core Server
Component/s: Queryable Encryption
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: Erwin Pe Assignee: Davis Haupt (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.2, v6.1, v6.0
Steps To Reproduce:

const kms = { key: BinData(0, "/tu9jUCBqZdwCelwE/EAm/4WqdxrSMi04B8e9uAV+m30rI1J2nhKZZtQjdvsSCwuI4erR6IEcEK+5eGUAODv43NDNIR9QheT2edWFewUfHKsl9cnzTc86meIzOmYl6dr")};
 
var testdb = db.getSiblingDB("testdb");
testdb.dropDatabase();
 
const csfleOpts = {
    kmsProviders: {
        local: kms,
    },
    keyVaultNamespace: "testdb.keystore",
    schemaMap: {},
};
 
var shell = Mongo(db.getMongo().host.toString(), csfleOpts);
var kv = shell.getKeyVault();
 
const schema = {
    "fields": [
        {
            "path": "foo",
            "keyId": kv.createKey("local", "ignored"),
            "bsonType": "string",
            "queries": {"queryType": "equality"}
        },
    ]
};
 
var edb = shell.getDB("testdb");
edb.createCollection("testcoll", {encryptedFields: schema});
 
var ecoll = edb.getCollection("testcoll");
ecoll.createIndex({__safeContent__: 1});
 
ecoll.insertOne({foo: "foovalue"});
 
ecoll.updateOne(
	{ $and: [{foo: "foovalue"}, {bar: "barvalue"}]},
	{ $set: { foo: "other_foovalue", bar: "other_barvalue" } },
	{ upsert: true }
)
 

Sprint: QO 2022-12-12, Security 2022-11-28
Participants:

 Description   

Suppose foo is a QE equality indexed field, bar is a unencrypted field, and testcoll is an encrypted collection containing a single document: {foo: "foovalue"}.

Running this command with upsert:

db.testcoll.updateOne(
	{ $and: [{foo: "foovalue"}, {bar: "barvalue"}]},
	{ $set: { foo: "other_foovalue", bar: "other_barvalue" } },
	{ upsert: true }
)

fails with the error message: Plan executor error during findAndModify :: caused by :: The field '_safeContent_' must be an array but is of type binData in document {no id}



 Comments   
Comment by Githook User [ 01/Dec/22 ]

Author:

{'name': 'Davis Haupt', 'email': 'davis.haupt@mongodb.com', 'username': 'davish'}

Message: SERVER-70134 fix fle2 upsert with single tag
Branch: master
https://github.com/mongodb/mongo/commit/3a5730f04cea8bef533c477f510ce9164bf1bd4d

Comment by Githook User [ 01/Dec/22 ]

Author:

{'name': 'Davis Haupt', 'email': 'davis.haupt@mongodb.com', 'username': 'davish'}

Message: SERVER-70134 fix fle2 upsert with single tag
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/3c9c4f3f1a91ea0b63add2271df4df7d23dcfde9

Comment by Davis Haupt (Inactive) [ 30/Nov/22 ]

The solution in the PR was to wrap all {_safeContent_: {$in: [...]}} predicates in an $elemMatch. All the queries are still executed the same due to the Single Query Condition, but upsert now knows that the single element should be in an array.

Comment by Erwin Pe [ 28/Nov/22 ]

Since this requires the query rewrite to handle the special case of upserts with 1 entry in a $in filter by rewriting it to an elemMatch, passing this ticket back to Query team.

Comment by Davis Haupt (Inactive) [ 28/Nov/22 ]

This seems related to SERVER-34973, which was closed as "works as designed". Asya had a suggestion in that ticket for how to rewrite the query to work in that scenario.

I don't think this is considered a bug in the query system, but just a unfortunate side effect of how arrays are queried in MQL. The solution would probably be to send a different upsert command to the query system with an $elemMatch rather than a $in if there is exactly one tag.

Comment by Erwin Pe [ 22/Nov/22 ]

davis.haupt@mongodb.com The $in query on _safeContent_ is being written correctly in fle_crud.cpp as the following debug log output shows:

{
    "t": {
        "$date": "2022-11-22T22:28:00.591+00:00"
    },
    "s": "D1",
    "c": "COMMAND",
    "id": 21962,
    "ctx": "FLECrud-19",
    "msg": "Assertion while executing command",
    "attr": {
        "command": "findAndModify",
        "db": "testdb",
        "commandArgs": {
            "findAndModify": "testcoll",
            "query": {
                "$and": [
                    {
                        "__safeContent__": {
                            "$in": [
                                {
                                    "$binary": {
                                        "base64": "7ELUSyGcnr/PEAjxgK6QAZgwwMXiCuyniXVQrtd+ABo=",
                                        "subType": "0"
                                    }
                                }
                            ]
                        }
                    },
                    {
                        "bar": {
                            "$eq": "barvalue"
                        }
                    }
                ]
            },
            "batchSize": 1,
            "singleBatch": true,
            "remove": false,
            "update": {
                "$set": {
                    "bar": "other_barvalue",
                    "foo": "###"
                },
                "$push": {
                    "__safeContent__": {
                        "$each": [
                            {
                                "$binary": {
                                    "base64": "wlyIMBj5C3nxr0h7sZvENtdZaNFi6t1LjjXF3FVQlnM=",
                                    "subType": "0"
                                }
                            }
                        ]
                    }
                }
            },
            "upsert": true,
            "stmtId": 2,
            "bypassDocumentValidation": false,
            "encryptionInformation": {
                "type": 1,
                "deleteTokens": {
                    "testdb.testcoll": {
                        "foo": {
                            "o": {
                                "$binary": {
                                    "base64": "X+vvpmLaveKqo63JIVz0NlsUyE2NJqmbpZlgsPsAU7I=",
                                    "subType": "0"
                                }
                            },
                            "e": {
                                "$binary": {
                                    "base64": "SiONVzq5rcYnFIrmCXqSBUVOOUhU6q5QWDOumVKO93w=",
                                    "subType": "0"
                                }
                            }
                        }
                    }
                },
                "schema": {
                    "testdb.testcoll": {
                        "escCollection": "enxcol_.testcoll.esc",
                        "eccCollection": "enxcol_.testcoll.ecc",
                        "ecocCollection": "enxcol_.testcoll.ecoc",
                        "fields": [
                            {
                                "keyId": {
                                    "$uuid": "ae05577e-af8e-4dd2-a905-0554b9118791"
                                },
                                "path": "foo",
                                "bsonType": "string",
                                "queries": {
                                    "queryType": "equality",
                                    "contention": 4
                                }
                            }
                        ]
                    }
                },
                "crudProcessed": true
            },
            "lsid": {
                "id": {
                    "$uuid": "2b8ad541-bee6-4fe5-b467-06d871cf0819"
                },
                "uid": {
                    "$binary": {
                        "base64": "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=",
                        "subType": "0"
                    }
                },
                "txnUUID": {
                    "$uuid": "4576774a-02f3-45a0-83e5-4e67e2271cc2"
                }
            },
            "txnNumber": 2,
            "autocommit": false,
            "$db": "testdb"
        },
        "error": "BadValue: Plan executor error during findAndModify :: caused by :: The field '__safeContent__' must be an array but is of type binData in document {no id}"
    }
}

Doing a findOne on the collection also shows the _safeContent_ as an array, so this issue is not coming from fle_crud.cpp.

My hypothesis of what's happening during the command goes as follows:

  1. updateOne gets rewritten to findAndModify with a single tag in the $in array (as above).
  2. the findAndModify query is evaluated, but doesn't match any documents.
  3. since upsert is true and no documents matched, the command proceeds with creating a new document. This document would initially contain the fields and values that were included in the query filter's equality operators. Since _safeContent is in the query filter, and since it only has one element in the $in array, this is effectively treated as an equality operation; hence, safeContent_ is added to the new document with the value being the bindata tag in the $in array.
  4. The operations in the update field of the command are then applied over this new document.  The $set works just fine, but the $push would throw an assertion because the _safeContent_ field value is a bindata, not an array.  This assertion happens here.
Comment by Erwin Pe [ 14/Oct/22 ]

davis.haupt@mongodb.com here you go:

const kms = { key: BinData(0, "/tu9jUCBqZdwCelwE/EAm/4WqdxrSMi04B8e9uAV+m30rI1J2nhKZZtQjdvsSCwuI4erR6IEcEK+5eGUAODv43NDNIR9QheT2edWFewUfHKsl9cnzTc86meIzOmYl6dr")};
 
var testdb = db.getSiblingDB("testdb");
testdb.dropDatabase();
 
const csfleOpts = {
    kmsProviders: {
        local: kms,
    },
    keyVaultNamespace: "testdb.keystore",
    schemaMap: {},
};
 
var shell = Mongo(db.getMongo().host.toString(), csfleOpts);
var kv = shell.getKeyVault();
 
const schema = {
    "fields": [
        {
            "path": "foo",
            "keyId": kv.createKey("local", "ignored"),
            "bsonType": "string",
            "queries": {"queryType": "equality"}
        },
    ]
};
 
var edb = shell.getDB("testdb");
edb.createCollection("testcoll", {encryptedFields: schema});
 
var ecoll = edb.getCollection("testcoll");
ecoll.createIndex({__safeContent__: 1});
 
ecoll.insertOne({foo: "foovalue"});
 
ecoll.updateOne(
	{ $and: [{foo: "foovalue"}, {bar: "barvalue"}]},
	{ $set: { foo: "other_foovalue", bar: "other_barvalue" } },
	{ upsert: true }
)
 

Generated at Thu Feb 08 06:15:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.