Capture pre-images for ONLY specified change events

XMLWordPrintableJSON

    • Query Execution
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Currently, pre-images are captured for all updates, replacements and deletes on a given collection.

      Ask: Enable capture of pre-images for ['DELETED', ...] documents only.
      **

      Context: A customer uses a collection for ingestion, where 16-20M documents are ingested every 20 minutes. Documents are indexed with Elastic Search upon insertion, and are removed from Elastic Search upon deletion. However, the ingestion process requires subsequent operations: an average of 30 updates are made on each document, and the customer doesn't want to pay the penalty of generating pre- and post-images on updates.
       

      Impact: Updating a collection of 80GB (20M docs of 4kb each) 30 times per document unnecessarily creates a Change Stream storage requirement of 2.4TB. This prevents the customer's adoption of full document change streams.
       

              Assignee:
              Unassigned
              Reporter:
              Matthew Royal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: