-
Type:
Investigation
-
Resolution: Won't Do
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Tools and Replicator
Currently, a $changeStream will error if an individual change event exceeds 16 MiB.
After this project, there will be a new opt in stage to split this large event into a sequence of fragments
The documentation should be updated to reflect this, and also explain how large events will show up in the stream to end users of the MongoDB Server manual.
Description of Linked Ticket
Summary
Customers using Change Streams (or MongoDB products on top of change streams) will no longer experience unexpected errors when using large documents.
Motivation
A document that is created as a result of a change stream document contains modification info, metadata, and pre- and post- images. However, a problem exists today for MongoDB customers: the document size limit in MongoDB is 16MB and some combination of the components of the change stream event (pre-image, post-image, update description) can result in exceeding the document size limit. Often, the pre-image or the post-image, and the change itself - need to be read at once, and it is very easy for the sum of these components to exceed 16MB.
The current behavior in this situation is for MongoDB to throw a BSONObjectTooLarge error, and the existing change stream is closed, requiring a new change stream to be opened.
This problem will impact more users soon - starting in 6.0, we will release the customer-facing version of this feature: User-facing Point-in-Time Pre and Post images (PM-1944). Here, users themselves will be able to retrieve both the pre- and post- images of a changed document in real time.
Documentation
Product Description
Scope Document
Technical Design Document