Add ability to publish full document only in changestream $source operator

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Critical - P2
    • 7.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • Sprint 34, Sprint 35, Sprint 36
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Most of the time, users who will be running stream queries will want to use the fullDocument of the change stream and not any metadata of the change stream event itself.  As it is today, users who use MongoDB as a source will have to to change their code to add a stage in order to remove the extra Change Stream metadata

      $replaceRoot: { newRoot : "$fullDocument"}

      and modify their timeField as follows:

      timeField : { $dateFromString :

      { "dateString" : "$fullDocument.timestamp"}

      },

      A better user experience would be to add a single configuration option as we have in Kafka Connector, publish.full.document.only. that would return just the value of the fullDocument of the Change Stream Event.

      The task:

      See some related documentation in the Kafka connector below:

      publish.full.document.only Type: boolean
       
      Description:
      Whether to return only the fullDocument field from the change stream event document produced by any update event. The fullDocument field contains the most current version of the updated document. To learn more about the fullDocument field, see the Server manual page on update events.
      When set to true, the connector overrides the change.stream.full.document setting and sets it to updateLookup so that the fullDocument field contains updated documents.
      Defaultfalse
      Accepted Valuestrue or false

       
       

            Assignee:
            Harendra Chawla
            Reporter:
            Robert Walters (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: