[SERVER-80909] Add ability to publish full document only in changestream $source operator Created: 12/Jun/23 Updated: 14/Nov/23 Resolved: 14/Nov/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.2.0-rc0 |
| Type: | Task | Priority: | Critical - P2 |
| Reporter: | Robert Walters | Assignee: | Harendra Chawla |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | init-337-m3, neweng | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Atlas Streams
|
| Backwards Compatibility: | Fully Compatible |
| Sprint: | Sprint 34, Sprint 35, Sprint 36 |
| Participants: |
| Description |
|
Most of the time, users who will be running stream queries will want to use the fullDocument of the change stream and not any metadata of the change stream event itself. As it is today, users who use MongoDB as a source will have to to change their code to add a stage in order to remove the extra Change Stream metadata $replaceRoot: { newRoot : "$fullDocument"} and modify their timeField as follows: timeField : { $dateFromString : { "dateString" : "$fullDocument.timestamp"}}, A better user experience would be to add a single configuration option as we have in Kafka Connector, publish.full.document.only. that would return just the value of the fullDocument of the Change Stream Event. The task:
See some related documentation in the Kafka connector below:
|
| Comments |
| Comment by Sandeep Dhoot [ 03/Aug/23 ] |
|
+1 for getting this done. It's annoying to have to specify `fullDocument.` anywhere including in the `timeField`. |
| Comment by Matthew Normyle [ 12/Jun/23 ] |
|
Also, how would this appear in the syntax? ex. $source: { config: { publish.full.document.only = true|false }} |
| Comment by Matthew Normyle [ 12/Jun/23 ] |
|
robert.walters@mongodb.com , when publish.full.document.only=true, is the expected output equivalent to $replaceRoot: { newRoot : "$fullDocument"} on the full changestream? |