[SERVER-69946] QE upsert excludes encrypted field in filter Created: 23/Sep/22  Updated: 05/Dec/22

Status: Backlog
Project: Core Server
Component/s: Queryable Encryption
Affects Version/s: 6.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kevin Albertson Assignee: Backlog - Security Team
Resolution: Unresolved Votes: 0
Labels: buildfest-2022
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
Assigned Teams:
Server Security
Operating System: ALL
Steps To Reproduce:

A full repro is included here: https://github.com/kevinAlbs/go-bootstrap/blob/418081b2e33688c8d88f47a1fecb75beceefc042/csfle/qe_updateOne/main.go

It runs an UpdateOne operation to upsert a new document. A Queryable Encryption (QE) field "encryptedIndexed" is included in the filter.

filter := bson.M{"_id": 1, "encryptedIndexed": "foo"}
update := bson.M{"$set": bson.M{"foo": "bar"}}
opts := options.Update().SetUpsert(true)
_, err = encryptedColl.UpdateOne(context.Background(),
	filter,
	update,
	opts)

Results in an upsert of the document:

{"_id":{"$numberInt":"1"},"foo":"bar"}

The expectation is for the document to include "encryptedIndexed".

The full update command sent is the following:

{
	"update": "encrypted",
	"ordered": true,
	"updates": [
	  {
		"q": {
		  "$and": [
			{
			  "_id": {
				"$eq": {
				  "$numberInt": "1"
				}
			  }
			},
			{
			  "encryptedIndexed": {
				"$eq": {
				  "$binary": {
					"base64": "BbEAAAAFZAAgAAAAAL7xGjpBxVi9AIH0MXbRGg9CYPzd42r9XjYcg8n5x9+zBXMAIAAAAAAMK7nVUWqB0CwpzVUMD4wmLn8yiKFvR9blOXKsSiHmRQVjACAAAAAActpJSZdO05aQQMFx5GQnrJcFbWFSx920vVpl6EebLqEFZQAgAAAAAE+1+k92587YImwuJWd/+080s+4IJmCobZoQjRHTTk8xEmNtAAAAAAAAAAAAAA==",
					"subType": "06"
				  }
				}
			  }
			}
		  ]
		},
		"u": {
		  "$set": {
			"foo": "bar"
		  }
		},
		"multi": false,
		"upsert": true
	  }
	],
	"encryptionInformation": {
	  "type": {
		"$numberInt": "1"
	  },
	  "schema": {
		"db.encrypted": {
		  "escCollection": "enxcol_.encrypted.esc",
		  "eccCollection": "enxcol_.encrypted.ecc",
		  "ecocCollection": "enxcol_.encrypted.ecoc",
		  "fields": [
			{
			  "keyId": {
				"$binary": {
				  "base64": "Pn4gGwr/Rq638KyxGcBpoA==",
				  "subType": "04"
				}
			  },
			  "path": "encryptedIndexed",
			  "bsonType": "string",
			  "queries": {
				"queryType": "equality",
				"contention": {
				  "$numberLong": "0"
				}
			  }
			}
		  ]
		}
	  },
	  "deleteTokens": {
		"db.encrypted": {
		  "encryptedIndexed": {
			"e": {
			  "$binary": {
				"base64": "T7X6T3bnztgibC4lZ3/7TzSz7ggmYKhtmhCNEdNOTzE=",
				"subType": "00"
			  }
			},
			"o": {
			  "$binary": {
				"base64": "wVFp0gQ3odC1Mohr8UQ/RRlYrAjnu8kiXPgsawE0wpU=",
				"subType": "00"
			  }
			}
		  }
		}
	  }
	},
	"lsid": {
	  "id": {
		"$binary": {
		  "base64": "hn5Jvj/zQEezCP8t5NbIGQ==",
		  "subType": "04"
		}
	  }
	},
	"txnNumber": {
	  "$numberLong": "1"
	},
	"$clusterTime": {
	  "clusterTime": {
		"$timestamp": {
		  "t": 1663944792,
		  "i": 3
		}
	  },
	  "signature": {
		"hash": {
		  "$binary": {
			"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=",
			"subType": "00"
		  }
		},
		"keyId": {
		  "$numberLong": "0"
		}
	  }
	},
	"$db": "db"
  }

Sprint: Security 2022-10-03, Security 2022-10-17
Participants:

 Description   

Upserting a document to a collection with Queryable Encryption excludes the encrypted field from the filter.

For example:

filter := bson.M{"_id": 1, "encryptedIndexed": "foo"}
update := bson.M{"$set": bson.M{"foo": "bar"}}
opts := options.Update().SetUpsert(true)
_, err = encryptedColl.UpdateOne(context.Background(),
	filter,
	update,
	opts)

Results in an upsert of the document:

{"_id":{"$numberInt":"1"},"foo":"bar"}



 Comments   
Comment by Sergey Galtsev (Inactive) [ 07/Oct/22 ]

Upsert behavior can be logically represented as a transaction:

for record in $(find where [id] == "foobar"):
   record.field1 = "foo"
   record.field2 = "bar"
   count++
 
if count == 0:
   record = $(create new)
   record.id = "foobar"
   record.field1 = "foo"
   record.field2 = "bar"

Expected behavior with upsert, where fields from filter are added to the "$set" is implemented on the server by taking a filter expression and transforming it into an insert. Notably, fields from filter are only applied when upsert results in an insert, not update.

With FLE2, the fields in the filter are not transferred verbatim. Instead, they are converted to crypto material, which could be used to find required record. This material can not be used to gerenate an insert.

There are two ways to implement the expected behavior:

1. have query analysis transform

filter = { id: 10}, set: { name: 'alice' }

into

filter = { id: 10}, set: { id: 10, name: 'alice' }

before involving FLE2 logic. That means however, that id would be always modified, even when upsert results in an update instead of an insert, which could cause a leakage

2. a protocol change could be made to send additional element (something like: $set-on-insert) in update protocol. The data sent would become (conceptually):

filter = { id: 10}, set: { name: 'alice' }, set-on-insert: { id: 10 }

At this time, my recommendation would be to document behavior as a known limitation of FLE2. If required, customers could emulate this behavior either by splitting upsert transaction into update/insert in their code (e.g.: manually implementing approach #2), or supply the search field twice - in filter and in update (e.g: implementing approach #1)

Comment by Sergey Galtsev (Inactive) [ 27/Sep/22 ]

To reproduce this behavior in a shell, use following code:

load("jstests/fle2/libs/encrypted_client_util.js");
 
const dbName = 'upsert_test';
const dbTest = db.getSiblingDB(dbName);
dbTest.dropDatabase();
 
const client = new EncryptedClient(db.getMongo(), dbName);
client.createEncryptionCollection("basic", {
    encryptedFields:
        { "fields": [
            { 
                "path": "encryptedIndexed",
                "bsonType": "string",
                "queries": { 
                    "queryType": "equality", 
                    "contention": 0
                }
            }
        ]}
});
 
const edb = client.getDB();
edb.basic.updateOne(
    { notEncrypted: "foo", encryptedIndexed: "bar" },
    { $set: { something: "foobar" } },
    { upsert: true });
 
jsTest.log("EDC: " + tojson(dbTest.basic.find().toArray()));

Upserted document will look as follows:

[js_test:bfm] [jsTest] ----
[js_test:bfm] [jsTest] EDC: [
[js_test:bfm] [jsTest] 	{
[js_test:bfm] [jsTest] 		"_id" : ObjectId("63331eaefe0adc21097a5aee"),
[js_test:bfm] [jsTest] 		"notEncrypted" : "foo",
[js_test:bfm] [jsTest] 		"something" : "foobar"
[js_test:bfm] [jsTest] 	}
[js_test:bfm] [jsTest] ]
[js_test:bfm] [jsTest] ----

According to the following link:

If no document matches the query criteria and the <update> parameter is a document with update operator expressions, then the operation creates a base document from the equality clauses in the <query> parameter and applies the expressions from the <update> parameter.

In other words, because encryptedIndexed: "bar" is in filter, it should also be upserted, but it is not

Generated at Thu Feb 08 06:14:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.