[SERVER-78739] Add ability for operations to be annotated with something that can be seen and filtered in change stream Created: 06/Jul/23  Updated: 27/Oct/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Tyler Kaye Assignee: Katya Kamenieva
Resolution: Unresolved Votes: 0
Labels: change-streams-support, changestreams, query-product-scope-3, query-product-value-2, roadmap
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Execution
Participants:
Case:

 Description   

More and more users and internal applications are utilizing change streams to replicate changes to/from other systems. For these systems, providing an easy way to not read your own writes would alleviate this currently difficult challenge.

Device Sync and Triggers make a lot of writes to MongoDB and also listen to all events in MongoDB. For Device Sync it is important that we filter out all writes that WE made. We do this by keeping track of all of the SessionIds that we use to make changes to MongoDB within a transaction and filter them out there (because they are in the change event)

This is cumbersome and a performance hit (keeping track of these sessions and persisting them).

In an ideal world, we could have some form of $comment that adds an annotation to an oplog entry that can be filtered out with a change stream match expression.

This could also be helpful for the triggers product, cluster-to-cluster replication, etc in an attempt to not read your own writes (or at least easily know that they are yours) and prevent infinitely recursive triggers (trigger that updates the collection it is listening to suddenly spiraling out of control)

Thread about this: https://mongodb.slack.com/archives/C6WSFCJDC/p1680026502600029



 Comments   
Comment by Daphne Levy [ 13/Sep/23 ]

Hi Team, I am sharing here a use case from a customer. The customer would like to have the possibility to know if a change stream delete event is triggered by Online Archive. The context is that they want to archive the data from the MongoDB cluster only without affecting their data warehouse which syncs data using the change stream API.

Generated at Thu Feb 08 06:39:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.