[SERVER-31720] Provide a way to get notification when results of a query change Created: 25/Oct/17 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Querying |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | NOVALUE Mitar | Assignee: | Backlog - Query Execution |
| Resolution: | Unresolved | Votes: | 8 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Execution
|
||||||||
| Participants: | |||||||||
| Description |
|
If one wants to observe changes to a set of documents matching a query, you can use fullDocument to match on document's content and get notifications for changes made to those documents. But this is limited in use by the fact that there is no notification when a document stops being matched by the query. See https://github.com/meteor/meteor-feature-requests/issues/158#issuecomment-339376181 of the issue as we are investigating how to use Change Streams to support reactivity in Meteor. |
| Comments |
| Comment by Asya Kamsky [ 17/Dec/18 ] |
|
I've decided to close the old ticket in favor of this one (as a duplicate) because this one has more context that current and relevant to the discussion of this feature, and leave this ticket as the one tracking the work that would be necessary to solve this request. |
| Comment by NOVALUE Mitar [ 06/Dec/18 ] |
|
Seems like the same feature request, but yes, now things changed much more so this could be seen only as a feature request on top of change streams, or materialized views. @asya: Sorry for not replying back, but there is really no set of queries one would be interested in. Frameworks like Meteor allow one to observe any type of queries in a reactive manner. So the question is how we could implement this natively using MongoDB instead of having to observe oplog or use change streams but then still cache all documents from the query in the server-side of the app as well, to detect deletions, reorderings and other aspects of changes to the query which is not really possible currently with change streams. Ideally, this would all be provided by MongoDB who has much better grasp of the query, indexes used, and knows well when something changed. |
| Comment by Charlie Swanson [ 06/Dec/18 ] |
|
Looks like a duplicate to me! That old one could use some of the references to change streams here - I think they're helpful for the context of why this isn't just change streams. |
| Comment by Asya Kamsky [ 06/Dec/18 ] |
|
I think this is a duplicate of
|
| Comment by Asya Kamsky [ 14/Sep/18 ] |
|
mitar sorry, I missed your question earlier. If you could provide several examples of such queries so I can think about whether there are efficiency issues when trying to address them, that would help.
|
| Comment by NOVALUE Mitar [ 09/Apr/18 ] |
|
But would those materialized views be scalable? I think current main issue is that change streams are already limited in number of concurrent streams you can open. Now if for each query one would have to make a materialized view and then change stream on top of that view, this could become a hit on performance. |
| Comment by Asya Kamsky [ 29/Mar/18 ] |
|
There is not a way to do this because that was not an intended use case for change streams. We hope to implement support for materialized views some time in the future and are keeping this use case in mind - so one possibility is it will be possible to watch for changes on materialized view (which will report inserts when documents start matching the query and deletes when documents stop matching). This is not something change streams the way it's design now can do. I'm going to convert this ticket to a new feature request to have this (or equivalent) functionality. |
| Comment by NOVALUE Mitar [ 28/Mar/18 ] |
|
I think the issue here is that we hope to use MongoDB Change Stream to get changes for a particular (and all types of) Mongo queries. Currently the easiest way to do so is by putting the query into $match pipeline against full document. We would not to have to parse queries ourselves and convert it in a way you are mentioning above. I am also not sure if more complicated queries can really be converted in this way. Ideally, API would be something like collection.find( {...}).watch() to start getting all changes for documents matching a query and when they stop matching a query. Is there a simple way to match this to the existing API? I do not see it. |
| Comment by Asya Kamsky [ 24/Mar/18 ] |
|
I think the solution should be in one change stream but it should be watching the mutable field for any changes, which for an update operation means matching if it's in the $set document (with $exists:true) or whether it's in the $unset array (so that document can be removed when the field is unset as it will no longer match whatever the value/threshold is). In other words, rather than watching for {x:{$gt:5} and {x:{$lte:5}} just watch for {x:{$exists:true}} in $set but if x can be unset also include $unset:"x" unoined (via $or) in the same match condition. |
| Comment by NOVALUE Mitar [ 23/Mar/18 ] |
|
Ah, yes. Sorry. To my understanding, we keep around the cache of preimages. We can then also notify users of a change between old and new value. But for this API, I am realizing that I think maybe the same thing could be achieved by opening two change stream requests. One with positive query and one with its negation. When a document ID appears in the second stream, you know it was removed from the first. So maybe this is something which could be internally handled by MongoDB in this way? |
| Comment by Kelsey Schubert [ 23/Mar/18 ] |
|
Hi mitar, We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket. Regards, |
| Comment by Alyson Cabral (Inactive) [ 13/Nov/17 ] |
|
Hi Mitar, You guys offer this functionality today by tailing the oplog, right? How do you do this? When an update happens to a document where {language : 'english'}--> {language : 'german'}the oplog only stores the new values not the values that were replaced. I imagine in order to notify on documents that no longer satisfy the query you must keep around a cache of pre-images with the old values. In order to do this on the core server, we would have to build something similar. Aly |
| Comment by NOVALUE Mitar [ 06/Nov/17 ] |
|
Thanks for the links. I think I have seen those. So our main use case is simply to reactively follow a query. But I must say that I do not understand why you would have to keep preimage around? If I make a query like {language: 'english'}The main issue from our experience comes when you also want to have sorting and limiting, but those basic queries should be pretty easy to detect? If there comes a change to language field which is not english, you know that you have to notify a listener that _id got removed. If change is to english, you notify that it has been added. |
| Comment by Alyson Cabral (Inactive) [ 26/Oct/17 ] |
|
Hi Mitar, I'm updating the type of this ticket to an 'improvement' as opposed to a 'bug'. We definitely want to enable seamless and straightforward ways to specify active queries in future versions of MongoDB. However, the 3.6 change streams design favors the resumability of change streams over this functionality. I want to point out that there are ways to satisfy these use cases with MongoDB 3.6, but we recognize that the approaches are not ideal. I linked my video presentation and Jesse's blog post below for further details (also linked on the github thread). In order to automatically notify listening applications once a query is no longer satisfied, we would need to keep around a pre-image in addition to the post-image of each operation you can resume from, either doubling the size or halving the operation capacity of the oplog. Another approach could be requiring some kind of pre registration of streams to minimize expense. While your desired functionality is not in the first version of Change Streams we are releasing, these are areas of feature development we are actively considering. We expect to learn a lot from the communities that adopt change streams and want work together on building out the feature development roadmap. Thanks for passing on thread to us. As you guys integrate with and develop against change streams, please reach out with any questions, concerns, or points of feedback. If you have specific examples of change stream use cases you want to support, I can help recommend the best approaches. Aly Cabral [1] Video: Using Change Streams to Keep Up with Your Data |