[SERVER-54844] changeStreams supports watch several dbs Created: 01/Mar/21 Updated: 29/Oct/23 Resolved: 11/Oct/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | None |
| Fix Version/s: | 5.1.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | vinllen chen | Assignee: | Backlog - Query Execution |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Execution
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Participants: | |||||||||
| Description |
|
Currently, changeStreams supports watching namespace by 3 granularities: single collection, single db, all dbs. If users want to monitor several dbs, we must watch all dbs and then use the filter stage or some outer method to only pass the namespace we want. However, this method is not efficient because there is so much unwanted data pass between mongos, mongod, and even client if we use the outer method to filter. In our multi-tenant severless environment, using "watch all dbs" will consume a lot of resources: db1, db2 belongs to user1; db3, db4 belongs to user2. So when we want to migrate user1 to another instance, we must start changeStreams to watch all dbs(db1, db2). |
| Comments |
| Comment by Bernard Gorman [ 11/Oct/21 ] |
|
As of release 5.1.0, $match and $project stages will be pushed down from mongoS to the shards where possible. In addition, $match filters will be rewritten such that they can be applied directly to the oplog, for greater efficiency in eliminating non-matching events. It is therefore possible to watch multiple DBs by opening a whole-cluster stream and filtering for something like {$match: {"ns.db": {$in: ["db1", "db2"]}}}. |
| Comment by Justin Seyster [ 17/Mar/21 ] |
|
Agreed, there is a substantial number of cases to consider for any kind of $match pushdown. For any future pushdown optimization we implement, we'll try to make it apply to as many scenarios as possible. This kind of feedback is helpful for us to understand which scenarios are useful to address with optimizations, so thanks for providing that additional background! |
| Comment by vinllen chen [ 17/Mar/21 ] |
|
Hi, Justin, thanks for your reply. Sorry for my previous answer made a mistake, watch all databases will consume a lot of resumes, but watch a single database won't. I agree with your idea that start some separate change stream to each database won't consume a lot of resources, however, the client needs to do the extra job to guarantee causal consistency between several databases, e.g., If user A has two databases: db1, db2. session1 that watching db1 receives event1(ts=1), event2(ts=10); session2 that watching db2 receives event3(ts=2), event4(ts=11). So the client needs to wait to sort by the timestamp(HLC) across two sessions: ts=1, ts=2, ts=10, ts=11, ... Actually, we modify the change stream code to fulfill our goals that watch several databases, we can commit our patch if you need. I think SERVER-48694 is a more general request. But in my opinion, it'll be so many cases to handle if push down all filter into mongod. I agree this can solve more common problems, looking forward to that. |
| Comment by Justin Seyster [ 16/Mar/21 ] |
|
Vinllen, thanks for the detailed description of your use case. You may find that having a separate change stream that is specific to each database is actually more efficient, because even though each change stream will have to read the entire oplog, they will each be able to operate on just the relevant subset of oplog entries, saving significantly on processing time and data transferred from shards to the mongos. It may be worth trying if you haven't already. I agree that being able to push the filter stage into the DocumentSourceOplogMatch would be a useful optimization and would be helpful in this situation. |
| Comment by Edwin Zhou [ 08/Mar/21 ] |
|
Thank you for the detailed example. We're assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen. Kind regards, |
| Comment by vinllen chen [ 06/Mar/21 ] |
|
Yes, only multiple changeStreams on the database level can fulfill our goals, however, this will consume a lot of resources because every changeStreams needs to fetch all oplog. So I think maybe it'll be better to support several database granularities or push down the aggregate $filter stage into the DocumentSourceOplogMatch stage, e.g., find all oplog, run the filter, and then run the following stage: DocumentSourceChangeStreamTransform, DocumentSourceCheckInvalidate, DocumentSourceShardCheckResumability, and so on. |
| Comment by Edwin Zhou [ 05/Mar/21 ] |
|
Thank you for submitting this feature improvement request. Have you found success opening multiple changeStreams on the database level to cover the databases you'd like to watch, rather than watching the entire deployment? Best, |