Investigate changes in SPM-2677: Avoiding Errors if Change Stream Events Exceed 16MB

XMLWordPrintableJSON

    • Type: Investigation
    • Resolution: Won't Do
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Tools and Replicator

      Original Downstream Change Summary

      Currently, a $changeStream will error if an individual change event exceeds 16 MiB.

      After this project, there will be a new opt in stage to split this large event into a sequence of fragments

      The documentation should be updated to reflect this, and also explain how large events will show up in the stream to end users of the MongoDB Server manual.

      Description of Linked Ticket

      Epic Summary

      Summary

      Customers using Change Streams (or MongoDB products on top of change streams) will no longer experience unexpected errors when using large documents.

      Motivation

      A document that is created as a result of a change stream document contains modification info, metadata, and pre- and post- images. However, a problem exists today for MongoDB customers: the document size limit in MongoDB is 16MB and some combination of the components of the change stream event (pre-image, post-image, update description) can result in exceeding the document size limit. Often, the pre-image or the post-image, and the change itself - need to be read at once, and it is very easy for the sum of these components to exceed 16MB. 

      The current behavior in this situation is for MongoDB to throw a BSONObjectTooLarge error, and the existing change stream is closed, requiring a new change stream to be opened. 

      This problem will impact more users soon - starting in 6.0, we will release the customer-facing version of this feature: User-facing Point-in-Time Pre and Post images (PM-1944). Here, users themselves will be able to retrieve both the pre- and post- images of a changed document in real time.

      Documentation

      Product Description
      Scope Document
      Technical Design Document

      POC Branch
      Docs Update

            Assignee:
            Jian Guan
            Reporter:
            Backlog - Core Eng Program Management Team
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: