Summary

The CRUD specifications do not clearly define how to batch documents for insert, delete, and update operations to handle cases where the resulting OP_MSG exceeds the maximum size of ~16MB, particularly when advancing past the first partition of the batch.

For example, when batching for an insert, the Go Driver partitions documents into ~16MB parts: p1, p2, p3, ... . If a part pi (where i > 1) fails with ErrDocumentTooLarge, then the operation terminates. In this case, partitions 1 through i - 1 would be inserted successfully, but the user would receive an error for partition i. In this case, should we still return a set of inserted IDs? Is this behavior generally acceptable?

Here is a repro using the Go Driver: https://gist.github.com/prestonvasquez/e28544554a872f4115389f1909f18ccd

Output:

2024/10/30 10:41:51 err:  an inserted document is too large
found:  100 

Motivation

Who is the affected end user?

Any drivers that validate CRUD batches as they are advanced for operation.

How does this affect the end user?

In the described scenario, users could insert a subset of the intended data before receiving an error noting that they are trying to insert a document that is too large for OP_MSG. 

How likely is it that this problem or use case will occur?

Unknown but definitely possible

If the problem does occur, what are the consequences and how severe are they?

If analogues to ErrDocumentTooLarge are not associated with a list of inserted IDs, such as the Go Driver behavior, it can be quite annoying to avoid duplicating data when re-attempting the write.

Is this issue urgent?

NA

Is this ticket required by a downstream team?

No

Is this ticket only for tests?

No

Acceptance Criteria

• Specify the limitations of OP_MSG in terms of insert, delete, or update. Specifically that documents must be batched into ~16mb partitions.
• Specify the behavior on what to do when a driver encounters a failure when trying to perform an insert, delete, or update on a subsequent (i > 1) partition.
  • Should we check all document sizes when advancing the initial batch and return an error if any of the documents exceed the max length? This might be considered a breaking change, but would ensure users aren't put in a strange position of inserting a subset of the documents they intended before getting an error.
  • Should we ensure that inserted ids are returned, if the driver has chosen to implement that optional behavior?
• Create unified spec tests modeling the decided behavior and (if its determined a bug) tests that try to insert a document that is too large after an initial subset of documents that total ~16 mb.