[SERVER-5042] Implement support for reliable change notifications Created: 22/Feb/12  Updated: 06/Dec/22  Resolved: 24/Jul/17

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Yuri Finkelstein Assignee: Backlog - Query Team (Inactive)
Resolution: Duplicate Votes: 28
Labels: pull-request
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-13932 Change Notification Stream API Closed
is duplicated by SERVER-19179 Database change notification Closed
is duplicated by SERVER-30252 Write oplog operations to kafka Closed
is duplicated by SERVER-13755 Support for tailable cursors over non... Closed
Related
related to SERVER-5730 Reliable notification framework Closed
is related to SERVER-13932 Change Notification Stream API Closed
is related to SERVER-3385 Publish/Subscribe (Message queue) fun... Closed
Assigned Teams:
Query
Participants:

 Description   

This features enables users to subscribe to change event stream (notifications) resulting from changes made to documents in a collection in mongodb. The user can specify a filter to be applied to change events. The resulting stream would go into a capped collection created specifically for this subscription. In addition, mongodb will keep a position in this capped collection reflecting the last message consumed by the client so that the client could resume reading the event stream where it left off.

Here is a more detailed version of how this should work:
1. Let's assume a mongodb has a collection called Item
2. User registers a subscription to events of interest in the collection Item
• A subscription filter is provided
• A destination capped collection name is specified, example: Item_changes; it serves as subscription queue
• The _id field in this collection contains a monotonously increasing sequence counter in the lower bits and date in the upper bits
o Host name, mac address, or any other such things are not present in the _id making it possible for the client to construct the _id by date only; this makes it possible for consumer of these records to indicate the starting point for scanning the notification records having just the starting date or the id the last record consumed, or nothing at all
3. Every time there is any mutation in collection Item, right after applying this mutation to the main collection (Item in this case) mongodb evaluates the list of all active subscriptions in Item, and if they match - appends a new record to corresponding destination capped collection.
4.When any of the mongodb clients call getLastError after submitting a modification to Item collection the system responds only after all subscriptions have been evaluated and corresponding notification events inserted into subscription queues and made durable with the same arguments as the current getLastError call.
• This is very important as it insures consistency of delivery of subscription events
5. Subscription life cycle management
• Create a subscription given a collection name and a filer; result - subscription ID
• List subscription details given a subscription ID
• Active/Deactivate subscription by ID
• Delete subscription by ID

This feature is basically very similar to how mongodb replication works.
The key differences are:
• subscription filter makes it possible to filer out only events of interest for a given consumer
• Instead of using this technique for replication only, mongodb suddenly becomes a source of customizable business events that are placed in durable queues and managed by mongo itself



 Comments   
Comment by Spencer Brody (Inactive) [ 24/Jul/17 ]

We are planning on releasing a change streams API in our upcoming 3.6 release. The exact implementation is a bit different than what is described in this ticket, but should provide similar functionality. The work for this project is spread out across multiple tickets, but you can follow SERVER-13932 to be notified when it has been completed.

Comment by ijammy [X] [ 02/Oct/14 ]

Also, I'm implementing something else, which is similar to what I did in the pull-request, but leveraging rabbitmq, an AMQP implementation.

Comment by ijammy [X] [ 02/Oct/14 ]

Thanks Benety & Paul

I don't know how the mongodb community like this feature enhancement (https://github.com/mongodb/mongo/pull/199) at this moment. but at Yottaa, we've been using this customized mongodb( https://github.com/ijammy/mongo ) in production to publish the changes to the subscribers who are interested in specific documents. and I also know a few people used the customized mongodb version as well.

The only issue is that every time we upgrade mongodb, I have to apply the patch to the new version.

So if there is a possibility to merge this kind of feature into the mainline, I would love to re-send the pull-request based on the mainline.
Ideas?

Comment by Paul Pedersen [ 09/May/13 ]

An interesting mongodb feature enhancement. I would like to revive the thread on this suggestion.

Comment by Scott Lyons [ 16/May/12 ]

My organization needs something just like this. So far I've been able to make use of https://github.com/mongodb/mongo/pull/199 with great effect.

Comment by Yuri Finkelstein [ 16/Apr/12 ]

Just to add bit more context as to why this is an important feature to have.

Consider a typical pattern: a business transaction is performed in a database and several off-line applications need to be notified of this event asynchronously. A common solution to this problem is for the application committing a transaction with the DB to also post a message to a pub/sub message broker.

Now, in reality this is easier said than done if certain degree of reliability is required. For one, how can one guarantee that notification event is never lost beween completion of DB transaction and posting it to the pub/sub broker? 2PC is rarely an option and Mongo siply does not support it.

Another issue is the need to manage throughput capacity of the message broker as the number of DB transaction is growing. Since capacity management in DB and message brokers are rarely performed by the same team (in a large org) this presents logistical challenges.

The feature proposed in this ticket would allow designers to completely eliminate the need for message broker in this pattern and reply on MongoDB for both business transactions and generation of related notifications. It solves both problems mentioned above nicely:
1. notification delivery guarantee would be provided by mongodb by means of getLastError
2. the message queues required to hold notifications would scale organically with mongodb cluster;

Additional consideration:
client can poll slaves to relieve master from this new burden
clients can use tailable cursors to interrogate event queues

Generated at Thu Feb 08 03:07:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.