[SERVER-13932] Change Notification Stream API Created: 13/May/14  Updated: 29/Jan/18  Resolved: 07/Dec/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.6.0

Type: New Feature Priority: Major - P3
Reporter: Osmar Olivo Assignee: Alyson Cabral (Inactive)
Resolution: Done Votes: 30
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-2150 Persistent Queries Closed
Duplicate
is duplicated by SERVER-5042 Implement support for reliable change... Closed
is duplicated by SERVER-30252 Write oplog operations to kafka Closed
is duplicated by SERVER-13755 Support for tailable cursors over non... Closed
is duplicated by SERVER-28708 Event handler in auditing functionality. Closed
Related
related to SERVER-5042 Implement support for reliable change... Closed
Backwards Compatibility: Fully Compatible
Participants:
Case:

 Description   

Currently the oplog can be used as a means to track changes on the system. You may want to do this for things such as detecting changes to determine cache invalidations.

Naturally, tailing the oplog is not an ideal medium for this as it's format can change at any time and MongoDB provides no guarantees when basing things off the oplog.

Furthermore, the oplog stores all events that occur on the system including internal events like removes and inserts from migrations that I may not want or may want to filter out.

Finally, the oplog is a mongod-wide component which under heavy traffic means I may be tailing much more data than I actually need if my application is focused on just one collection or DB.

Ideally, creating an activity stream that is optionally published out would expose this through an API layer independent of the oplog. This would allow for applications like this to no longer directly rely on the oplog.

If we could also filter on the activity stream similar to how audit filters are applied it would allow for only reading the changes you care about as oppose to filtering on the application side and transmitting a ton of data needlessly.

You could also do things like only allowing people with certain permissions to receive certain changes. I.E. if I don't have permission to a DB, I can't view the changes that have occurred on it. Currently if you want to tail some subset of the oplog, you have to be able to tail the whole oplog which causes a bit of a security permissions problem.



 Comments   
Comment by Alyson Cabral (Inactive) [ 06/Dec/17 ]

I'm happy to announce that MongoDB 3.6 launched with Change Streams.

As a new feature in MongoDB 3.6, change streams enable applications to stream real-time data changes by leveraging MongoDB’s underlying replication capabilities. The characteristics of change streams include:

1. Resumable
Resumability was top of mind when building out change streams to enable applications to confidently see every change in a collection. Each change stream response includes a resume token. In cases where a connection between the application and server is lost, the application can send up the last resume token it received and change streams will pick up right where the application left off as if there was no interruption at all. In cases of transient network errors or elections, the drivers will automatically make an attempt to reestablish a connection. The drivers automatically cache the last resume token seen. However, to prepare for cases of application failure, applications need to persist the resume token periodically, as drivers do not maintain state after application restarts.
2. Targeted changes
Changes can be filtered to provide relevant and targeted changes to listening applications. As an example, filters can be on operation type or fields within the document itself.
3. Total ordering of changes across entire cluster
MongoDB 3.6 has a global logical clock that enables the server to order all changes across the entire cluster. Applications will always receive changes in the order they were applied on the server.
4. Durable changes
Change streams only notify on majority committed changes. This means that every change seen by listening applications is durable in failure scenarios like a new primary being elected in the event of your existing primary going down.
5. All changes are idempotent
All changes are transformed into a format that’s safe to apply multiple times. Listening applications don’t have to resume exactly at the last change they saw because reapplying operations is safe.

For more detailed information check out the following resources:
Video: Using Change Streams to Keep Up with Your Data
Documentation: Change Streams
Course: M036

Aly Cabral
MongoDB Product Manager

Comment by Shane Tolmie [X] [ 29/Jul/14 ]

I am interested in this feature, as it will allow any service to receive notifications when a portion of the database is edited by a 3rd party (even a human being typing on the console). This sort of behavior is very useful in a Service Oriented Architecture.

SQL Server has already solved this problem by providing SQL Service Broker. We use this feature heavily, so a service can get notified when anything at all edits a central database. The service has to react to edits can be generated by anything, e.g. a human being typing manually on the MongoDB console or someone manually editing the database using RoboMongo.

Generated at Thu Feb 08 03:33:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.