[SERVER-80909] Add ability to publish full document only in changestream $source operator Created: 12/Jun/23  Updated: 14/Nov/23  Resolved: 14/Nov/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Task Priority: Critical - P2
Reporter: Robert Walters Assignee: Harendra Chawla
Resolution: Fixed Votes: 0
Labels: init-337-m3, neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Atlas Streams
Backwards Compatibility: Fully Compatible
Sprint: Sprint 34, Sprint 35, Sprint 36
Participants:

 Description   

Most of the time, users who will be running stream queries will want to use the fullDocument of the change stream and not any metadata of the change stream event itself.  As it is today, users who use MongoDB as a source will have to to change their code to add a stage in order to remove the extra Change Stream metadata

$replaceRoot: { newRoot : "$fullDocument"}

and modify their timeField as follows:

timeField : { $dateFromString :

{ "dateString" : "$fullDocument.timestamp"}

},

A better user experience would be to add a single configuration option as we have in Kafka Connector, publish.full.document.only. that would return just the value of the fullDocument of the Change Stream Event.

The task:

See some related documentation in the Kafka connector below:

publish.full.document.only Type: boolean
 
Description:
Whether to return only the fullDocument field from the change stream event document produced by any update event. The fullDocument field contains the most current version of the updated document. To learn more about the fullDocument field, see the Server manual page on update events.
When set to true, the connector overrides the change.stream.full.document setting and sets it to updateLookup so that the fullDocument field contains updated documents.
Defaultfalse
Accepted Valuestrue or false

 
 



 Comments   
Comment by Sandeep Dhoot [ 03/Aug/23 ]

+1 for getting this done. It's annoying to have to specify `fullDocument.` anywhere including in the `timeField`.

Comment by Matthew Normyle [ 12/Jun/23 ]

Also, how would this appear in the syntax? ex.

$source: {

   config:

{       publish.full.document.only = true|false    }

}

Comment by Matthew Normyle [ 12/Jun/23 ]

robert.walters@mongodb.com , when publish.full.document.only=true, is the expected output equivalent to $replaceRoot: { newRoot : "$fullDocument"}  on the full changestream?

Generated at Thu Feb 08 06:44:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.