[SERVER-40354] mongocryptd should echo back all fields in command, and none that aren't Created: 27/Mar/19  Updated: 29/Oct/23  Resolved: 06/May/19

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 4.1.11

Type: Improvement Priority: Major - P3
Reporter: Jeffrey Yemin Assignee: Ted Tuckman
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-58293 mongocryptd does not include server A... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2019-05-06, Query 2019-05-20
Participants:

 Description   

mongodcryptd should respect whatever fields the caller passes to it, and echo them back unchanged (except for jsonSchema field and the markings, of course), and should not add any additional fields. This gives drivers the most flexibility in terms of the design space for implementation of FLE. Some drivers may be invoking mongocryptd at a layer above where $db and other common fields are added, and some below. By echoing back what the driver provides, it will allow either design to work seamlessly.

Current behavior:

MongoDB Enterprise > db.runCommand({insert : "test", documents : [{}], jsonSchema : {properties : {}}})
{
	"hasEncryptionPlaceholders" : false,
	"result" : {
		"insert" : "test",
		"bypassDocumentValidation" : false,
		"ordered" : true,
		"documents" : [
			{
 
			}
		],
		"lsid" : {
			"id" : UUID("a990143c-585b-4be0-ab45-93427f476d12")
		},
		"$db" : "test"
	},
	"ok" : 1
}

Expected behavior:

MongoDB Enterprise > db.runCommand({insert : "test", documents : [{}], jsonSchema : {properties : {}}})
{
	"hasEncryptionPlaceholders" : false,
	"result" : {
		"insert" : "test",
		"documents" : [
			{
			}
		]
	},
	"ok" : 1
}



 Comments   
Comment by Githook User [ 06/May/19 ]

Author:

{'name': 'Ted Tuckman', 'username': 'TedTuckman', 'email': 'ted.tuckman@mongodb.com'}

Message: SERVER-40354 Remove $db from returned document
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/a257d05fb12c56862386e3bd9bcf1c2c2cc7e811

Comment by Ted Tuckman [ 24/Apr/19 ]

Wound up doing this by looking at the original command and removing all fields from the response that were not in the original just before it gets sent back to the drivers. This should work for all the fields discussed.

Comment by Githook User [ 24/Apr/19 ]

Author:

{'email': 'ted.tuckman@mongodb.com', 'name': 'Ted Tuckman', 'username': 'TedTuckman'}

Message: SERVER-40354 mongocryptd should echo back all fields in command and none that are not
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/ce89b307d2e341a6c8829018a12c94b8998486d4

Comment by Mark Benvenuto [ 28/Mar/19 ]

Here is the logic of what gets echoed back in the result field for a command in mongocryptd:

  1. The command name and the command's fields. Using insert as an example, the list of fields echoed back are as follows:
    1. insert
    2. ordered (default value: false)
    3. documents
    4. bypassDocumentValidation (default value: true)
    5. stmtId
    6. stmtIds
  2. Any "generic" arguments sent to the server. This means things like "$db", "lsid", etc that are common to all commands. See the complete list here

Other notes

  • $db - if "$db" is not sent to server, it will be defaulted to "admin". This defaulting happens in the server-side message processing layer before it gets to any query code.
  • "bypassDocumentValidation", "ordered", and other default valued fields. These fields are automatically defaulted by the insert parsing code (same applies to update and delete) which is why you see them in the output. Reference

 

For rule 1, insert/update/delete/distinct (IDL based commands) will always echo back values with default values set even if they were not sent by the client. This is the way the generated serializers work for these operations. For find/findAndModify/count/etc, I cannot speak to them. Changing this IDL behavior is difficult. If we needed to change it, they can be stripped at the cost of a new pass to strip fields that were not in the input before returning them to the client.

For rule 2, I thought it would be easiest to echo the fields back to the client that were sent to the client. This behavior can easily be changed (with the possible exception of $db in some cases).

 

Comment by Craig Homa [ 28/Mar/19 ]

The Query team will need to discuss this further with the Drivers teams prior to implementing. 

Comment by Jeffrey Yemin [ 27/Mar/19 ]

Proposal:

  • Strip $db, $readPreference, and jsonSchema
  • Leave all other fields unchanged, except insofar as necessary to mark encrypted fields.
  • Don't add any new fields (e.g. ordered : false or bypassDocumentValidation : false

CC kevin.albertson, matt.broadstone@mongodb.com, mark.benvenuto@mongodb.com

Comment by Jeffrey Yemin [ 27/Mar/19 ]

Here's what the POC driver is actually sending:

{
  "insert": "default",
  "ordered": true,
  "documents": [
    {
      "_id": 1,
      "ssn": "457-55-5462"
    }
  ],
  "jsonSchema": {
    "properties": {
      "ssn": {
        "encrypt": {
          "keyId": [
            {
              "$binary": {
                "base64": "AAAAAAAAAAAAAAAAAAAAAA==",
                "subType": "04"
              }
            }
          ],
          "bsonType": "string",
          "algorithm": "AEAD_AES_256_CBC_HMAC_SHA_512-Deterministic",
          "initializationVector": {
            "$binary": {
              "base64": "aWlpaWlpaWlpaWlpaWlpaQ==",
              "subType": "00"
            }
          }
        }
      }
    },
    "bsonType": "object"
  },
  "$db": "default",
  "$readPreference": {
    "mode": "primaryPreferred"
  }
}

and here's what mongocryptd is returning:

{
  "hasEncryptionPlaceholders": true,
  "result": {
    "insert": "default",
    "bypassDocumentValidation": false,
    "ordered": true,
    "documents": [
      {
        "_id": 1,
        "ssn": {
          "$binary": {
            "base64": "AH0AAAACYQAsAAAAQUVBRF9BRVNfMjU2X0NCQ19ITUFDX1NIQV81MTItRGV0ZXJtaW5pc3RpYwAFaXYAEAAAAABpaWlpaWlpaWlpaWlpaWlpBWtpABAAAAAEAAAAAAAAAAAAAAAAAAAAAAJ2AAwAAAA0NTctNTUtNTQ2MgAA",
            "subType": "06"
          }
        }
      }
    ],
    "$db": "default",
    "$readPreference": {
      "mode": "primaryPreferred"
    }
  },
  "ok": 1.0
}

mark.benvenuto@mongodb.com can you explain the algorithm for the generation of fields in the result document? I can understand why it's echoing $db and $readPreference, but why bypassDocumentValidation?

Comment by Jeffrey Yemin [ 27/Mar/19 ]

Thought about this a bit more:  

  • still think that mongocryptd should not be adding fields like bypassDocumentValidation or ordered
  • fields like $db, $readPreference, etc are naturally going to be added to the command by the driver as it sends it to mongocryptd.  I don't think we can avoid it.  So we're going to have to decide which fields should be echoed back and which should not.  Judging from my POC, it's more natural to insert FLE support at a layer above where these common fields are added, so I would prefer that they not be echoed back.  But the drivers team is going to have to discuss what's going to work best across the full range of driver implementations, and then make a decision.
Generated at Thu Feb 08 04:54:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.