-
Type:
Task
-
Resolution: Fixed
-
Priority:
Critical - P2
-
Affects Version/s: None
-
Component/s: None
-
Atlas Streams
-
Fully Compatible
-
Sprint 34, Sprint 35, Sprint 36
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Most of the time, users who will be running stream queries will want to use the fullDocument of the change stream and not any metadata of the change stream event itself. As it is today, users who use MongoDB as a source will have to to change their code to add a stage in order to remove the extra Change Stream metadata
$replaceRoot: { newRoot : "$fullDocument"}
and modify their timeField as follows:
timeField : { $dateFromString :
{ "dateString" : "$fullDocument.timestamp"}},
A better user experience would be to add a single configuration option as we have in Kafka Connector, publish.full.document.only. that would return just the value of the fullDocument of the Change Stream Event.
The task:
- Amend ChangeStreamSourceOperators to support a $source.fullDocumentOnly: bool field. See https://github.com/10gen/mongo-enterprise-modules/blob/master/src/streams/exec/stages.idl#L230 for the interface definition.
- When fullDocumentOnly is true, `fullDocument: updateLookup, whenAvailable, or required` must also be specified.
- When fullDocumentOnly is true, the ChangeStreamSourceOperator should project the `fullDocument` field from the change events. See https://github.com/10gen/mongo-enterprise-modules/blob/master/src/streams/exec/change_stream_source_operator.cpp#L347
- Amend this test suite to test the changes: https://github.com/10gen/mongo-enterprise-modules/blob/master/jstests/streams/change_stream_source.js
See some related documentation in the Kafka connector below:
publish.full.document.only | Type: boolean Description: Whether to return only the fullDocument field from the change stream event document produced by any update event. The fullDocument field contains the most current version of the updated document. To learn more about the fullDocument field, see the Server manual page on update events. When set to true, the connector overrides the change.stream.full.document setting and sets it to updateLookup so that the fullDocument field contains updated documents. Default: false Accepted Values: true or false |