Support joining change stream large events that were split

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      By default ASP supports handling large change stream events and it will automatically stitch the split fragments together to recreate the underlying event. 

       

      https://github.com/mongodb/mongo-kafka/pull/190 is a good starting point for how to structure the configuration. https://github.com/Calvinnix/mongo-kafka/tree/reassembleLargeEvents has an initial implementation but we want to be careful to consider all the new edge cases that come with keeping temporary state since we need to collect the event fragments from the changestream before combining them.

       

      Server version 6.0.9+ adds support for splitting large change stream events.

      If a change stream has large events that exceed 16 MB, a BSONObjectTooLarge exception is returned. Starting in MongoDB 6.0.9, you can use a $changeStreamSplitLargeEvent stage to split the events into smaller fragments.

      You should only use $changeStreamSplitLargeEvent when strictly necessary. For example, if your application requires full document pre or post-images, and generates large events that exceed 16 MB, use $changeStreamSplitLargeEvent.-

      See: $changeStreamSplitLargeEvent .

      Recommend adding a new configuration so users can opt in to support this feature.

            Assignee:
            Unassigned
            Reporter:
            Ross Lawley
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: