-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
(copied to CRM)
-
None
-
None
-
None
-
None
-
None
-
None
-
None
By default ASP supports handling large change stream events and it will automatically stitch the split fragments together to recreate the underlying event.
https://github.com/mongodb/mongo-kafka/pull/190 is a good starting point for how to structure the configuration. https://github.com/Calvinnix/mongo-kafka/tree/reassembleLargeEvents has an initial implementation but we want to be careful to consider all the new edge cases that come with keeping temporary state since we need to collect the event fragments from the changestream before combining them.
Server version 6.0.9+ adds support for splitting large change stream events.
If a change stream has large events that exceed 16 MB, a BSONObjectTooLarge exception is returned. Starting in MongoDB 6.0.9, you can use a $changeStreamSplitLargeEvent stage to split the events into smaller fragments.
You should only use $changeStreamSplitLargeEvent when strictly necessary. For example, if your application requires full document preor post-images, and generates large events that exceed 16 MB, use $changeStreamSplitLargeEvent.-
See: $changeStreamSplitLargeEvent .
Recommend adding a new configuration so users can opt in to support this feature.
- describes design of
-
KAFKA-247 Recreate change stream from the point of failure for event > 16 MB
-
- Closed
-
- is related to
-
SERVER-55062 Change stream events can exceed 16MB with no workaround
-
- Closed
-
-
KAFKA-247 Recreate change stream from the point of failure for event > 16 MB
-
- Closed
-