[SERVER-78739] Add ability for operations to be annotated with something that can be seen and filtered in change stream Created: 06/Jul/23 Updated: 27/Oct/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Tyler Kaye | Assignee: | Katya Kamenieva |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | change-streams-support, changestreams, query-product-scope-3, query-product-value-2, roadmap | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Query Execution
|
| Participants: | |
| Case: | (copied to CRM) |
| Description |
|
More and more users and internal applications are utilizing change streams to replicate changes to/from other systems. For these systems, providing an easy way to not read your own writes would alleviate this currently difficult challenge. Device Sync and Triggers make a lot of writes to MongoDB and also listen to all events in MongoDB. For Device Sync it is important that we filter out all writes that WE made. We do this by keeping track of all of the SessionIds that we use to make changes to MongoDB within a transaction and filter them out there (because they are in the change event) This is cumbersome and a performance hit (keeping track of these sessions and persisting them). In an ideal world, we could have some form of $comment that adds an annotation to an oplog entry that can be filtered out with a change stream match expression. This could also be helpful for the triggers product, cluster-to-cluster replication, etc in an attempt to not read your own writes (or at least easily know that they are yours) and prevent infinitely recursive triggers (trigger that updates the collection it is listening to suddenly spiraling out of control) Thread about this: https://mongodb.slack.com/archives/C6WSFCJDC/p1680026502600029 |
| Comments |
| Comment by Daphne Levy [ 13/Sep/23 ] |
|
Hi Team, I am sharing here a use case from a customer. The customer would like to have the possibility to know if a change stream delete event is triggered by Online Archive. The context is that they want to archive the data from the MongoDB cluster only without affecting their data warehouse which syncs data using the change stream API. |