Summary

Support auto-encryption with document sequences in libmongocrypt and drivers.

SERVER-40364 suggests mongocryptd already supports OP_MSG document sequence.

Consider requesting the the crypt_shared library support document sequences.

Motivation

I expect supporting a document sequence may benefit performance of auto encryption:

• May avoid copies of write payloads.
• May permit increasing the bulk write batch size for auto-encryption and reduce round trips.

Background

In the MongoDB wire protocol, a document sequence may be included as an OP_MSG payloadType=1 section as an optimization (documents are sent back-to-back, rather than nested in an array).

At present, the libmongocrypt and the crypt_shared components of In-Use Encryption (CSFLE/QE) do not support document sequences.

If a driver constructs a document sequence for a write, the document sequence must be converted to a BSON array prior to sending to libmongocrypt. Example:

Driver constructs a document sequence for an insert to be sent in an OP_MSG payloadType=1:

{ "_id": 1, "encrypted": "foo" }
{ "_id": 2, "encrypted": "bar" }

Automatic encryption does not support document sequence. The document sequence must be converted to a BSON array prior to passing to libmongocrypt:

{
    "insert": "coll",
    "documents": [
        { "_id": 1, "encrypted": "foo" },
        { "_id": 2, "encrypted": "bar" }
    ]
}

For example, the C driver has an internal function to conversion.

The maximum size of a payloadType=0 payload is ~16MiB. As a consequence, bulk writes for automatic encryption are split at a reduced batch size of 2MiB. The maximum size of a payloadType=1 payload is ~48MiB.

Who is the affected end user?

Users of QE/CSFLE.

How does this affect the end user?

This change may improve write performance.

How likely is it that this problem or use case will occur?

Likely. Suggest resolving DRIVERS-2669 first (Add performance benchmarks for FLE/QE) to help measure a proposed improvement.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

No.

Acceptance Criteria

• Update libmongocrypt to support document sequence for automatic encryption.
• Update drivers to support passing document sequences to libmongocrypt for automatic encryption.