[KAFKA-381] Support change stream split large events Created: 25/Jul/23  Updated: 14/Dec/23

Status: Backlog
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: 1.12.0

Type: Improvement Priority: Major - P3
Reporter: Ross Lawley Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Design
describes design of KAFKA-247 Recreate change stream from the point... Closed
Related
is related to SERVER-55062 Change stream events can exceed 16MB ... Closed
is related to KAFKA-247 Recreate change stream from the point... Closed
Quarter: FY24Q3
Case:

 Description   

Server version 6.0.9+ adds support for splitting large change stream events.

If a change stream has large events that exceed 16 MB, a BSONObjectTooLarge exception is returned. Starting in MongoDB 6.0.9, you can use a $changeStreamSplitLargeEvent stage to split the events into smaller fragments.

You should only use $changeStreamSplitLargeEvent when strictly necessary. For example, if your application requires full document pre- or post-images, and generates large events that exceed 16 MB, use $changeStreamSplitLargeEvent.

See: $changeStreamSplitLargeEvent .

Recommend adding a new configuration so users can opt in to support this feature.



 Comments   
Comment by Robert Walters [ 25/Jul/23 ]

Yes, multiple fragments works

Generated at Thu Feb 08 09:06:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.