[SERVER-46929] Parallel consumers(subscribers) for ChangeStream Created: 18/Mar/20  Updated: 21/Mar/23  Resolved: 25/Mar/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Atil Pai Assignee: Backlog - Triage Team
Resolution: Done Votes: 0
Labels: change-streams-improvements
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Server Triage
Participants:
Case:

 Description   

I am R&D'ing on currently leveraging Mongo ChangeStreams.

I am seeing that it has a good throughput, resilient to failures due to the 'Resume token', but I dont see any documentation on whether if/ how we can manage multiple instances of a subscriber application to receive changes in a load balance manner. Without this, we would have to use a single instance per filtered stream, inherently dealing with single point of failure.

If this is currently not offered by the mongo drivers (we use scala), it would be great help if you could direct us to some other implementations that could be done to achieve this.



 Comments   
Comment by Carl Champain (Inactive) [ 25/Mar/20 ]

Hi atil.pai.992@my.csun.edu,

If you are referring to:

  1. Multiple client applications each consuming an identical stream of change events
  2. Possibly with events being interleaved between clients? ("load balancing manner")

There are a couple of ways you could do this:

  • The obvious approach is to open one stream per application instance. Because each stream will open an independent cursor on MongoDB, however, this will gradually cause a performance impact if you open a very large number of concurrent client instances.
  • A better option might be to write a "proxy" application which is responsible for managing a single stream, and which is capable of publishing each event to multiple subscribers using something like RabbitMQ. You could deploy a fleet of N proxies in front of MongoDB, and then have their end-consumer application connect to the proxies. That way, you could have multiple redundant proxies for each distinct stream, while only needing N cursors on the MongoDB deployment to deliver events to N*M client applications.

That said, the SERVER project is for bugs and feature suggestions for the MongoDB server. As this ticket does not appear to be a bug, I will now close it. If you need assistance troubleshooting, I encourage you to ask our community by posting on the MongoDB Community Forums or on Stack Overflow with the mongodb tag.

Kind regards,
Carl

Generated at Thu Feb 08 05:12:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.