[SERVER-54844] changeStreams supports watch several dbs Created: 01/Mar/21  Updated: 29/Oct/23  Resolved: 11/Oct/21

Status: Closed
Project: Core Server
Component/s: Querying
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: vinllen chen Assignee: Backlog - Query Execution
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-48694 Push down user-defined stages in a ch... Closed
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Currently, changeStreams supports watching namespace by 3 granularities: single collection, single db, all dbs. If users want to monitor several dbs, we must watch all dbs and then use the filter stage or some outer method to only pass the namespace we want. However, this method is not efficient because there is so much unwanted data pass between mongos, mongod, and even client if we use the outer method to filter.

In our multi-tenant severless environment, using "watch all dbs" will consume a lot of resources: db1, db2 belongs to user1; db3, db4 belongs to user2. So when we want to migrate user1 to another instance, we must start changeStreams to watch all dbs(db1, db2).



 Comments   
Comment by Bernard Gorman [ 11/Oct/21 ]

As of release 5.1.0, $match and $project stages will be pushed down from mongoS to the shards where possible. In addition, $match filters will be rewritten such that they can be applied directly to the oplog, for greater efficiency in eliminating non-matching events. It is therefore possible to watch multiple DBs by opening a whole-cluster stream and filtering for something like {$match: {"ns.db": {$in: ["db1", "db2"]}}}.

Comment by Justin Seyster [ 17/Mar/21 ]

Agreed, there is a substantial number of cases to consider for any kind of $match pushdown. For any future pushdown optimization we implement, we'll try to make it apply to as many scenarios as possible. This kind of feedback is helpful for us to understand which scenarios are useful to address with optimizations, so thanks for providing that additional background!

Comment by vinllen chen [ 17/Mar/21 ]

Hi, Justin, thanks for your reply.

Sorry for my previous answer made a mistake, watch all databases will consume a lot of resumes, but watch a single database won't.

I agree with your idea that start some separate change stream to each database won't consume a lot of resources, however, the client needs to do the extra job to guarantee causal consistency between several databases, e.g., If user A has two databases: db1, db2. session1 that watching db1 receives event1(ts=1), event2(ts=10); session2 that watching db2 receives event3(ts=2), event4(ts=11). So the client needs to wait to sort by the timestamp(HLC) across two sessions: ts=1, ts=2, ts=10, ts=11, ...

Actually, we modify the change stream code to fulfill our goals that watch several databases, we can commit our patch if you need. I think SERVER-48694 is a more general request. But in my opinion, it'll be so many cases to handle if push down all filter into mongod. I agree this can solve more common problems, looking forward to that.

Comment by Justin Seyster [ 16/Mar/21 ]

Vinllen, thanks for the detailed description of your use case. You may find that having a separate change stream that is specific to each database is actually more efficient, because even though each change stream will have to read the entire oplog, they will each be able to operate on just the relevant subset of oplog entries, saving significantly on processing time and data transferred from shards to the mongos. It may be worth trying if you haven't already.

I agree that being able to push the filter stage into the DocumentSourceOplogMatch would be a useful optimization and would be helpful in this situation. SERVER-48694 discusses another use case where such an optimization would be useful, and it includes a helpful comment from bernard.gorman outlining some of the details involved in implementing the optimization. We will definitely consider this use case as part of the design for any upcoming change streams optimization work.

Comment by Edwin Zhou [ 08/Mar/21 ]

Hi cvinllen@gmail.com,

Thank you for the detailed example. We're assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen.

Kind regards,
Edwin

Comment by vinllen chen [ 06/Mar/21 ]

Yes, only multiple changeStreams on the database level can fulfill our goals, however, this will consume a lot of resources because every changeStreams needs to fetch all oplog. So I think maybe it'll be better to support several database granularities or push down the aggregate $filter stage into the DocumentSourceOplogMatch stage, e.g., find all oplog, run the filter, and then run the following stage: DocumentSourceChangeStreamTransform, DocumentSourceCheckInvalidate, DocumentSourceShardCheckResumability, and so on.

Comment by Edwin Zhou [ 05/Mar/21 ]

Hi cvinllen@gmail.com,

Thank you for submitting this feature improvement request. Have you found success opening multiple changeStreams on the database level to cover the databases you'd like to watch, rather than watching the entire deployment?

Best,
Edwin

Generated at Thu Feb 08 05:34:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.