[SERVER-21889] Oplog format for $set operations on subdocuments creates format that is not queryable Created: 14/Dec/15 Updated: 25/Oct/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Write Ops |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Lee Parayno | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 10 |
| Labels: | changestreams, query-44-grooming | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Query Optimization
|
||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Steps To Reproduce: | Create initial document:
Run same statement again to create an update in the oplog:
Query for the oplog entry:
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||
| Description |
|
When updating a document that uses a subdocument reference for the find parameters, an oplog entry is created that cannot be queried for specifically because the syntax of the document does not allow dots to be entered into keys for a document--for any normal inserts into collections. See the full context with some tests in the discussion here: |
| Comments |
| Comment by Mansoor Omar [ 25/Oct/23 ] | ||||||||||||||||||
|
Those who are experiencing this issue for Trigger Match expressions when trying to filter for changes to a field in a sub-document should be able to use the workaround below:
| ||||||||||||||||||
| Comment by Enrico Schaaf [ 05/Apr/22 ] | ||||||||||||||||||
|
Thank you for your fast response, we use MongoDB 5 so I can use $getField but I sadly don't know how to combine it with $type. Could you show me how to use it with an example? | ||||||||||||||||||
| Comment by Asya Kamsky [ 05/Apr/22 ] | ||||||||||||||||||
|
The workaround works with existence check - you should be checking $type of the field rather than value. Also if it’s 5.0 then you can use $getField expression to simplify. | ||||||||||||||||||
| Comment by Enrico Schaaf [ 05/Apr/22 ] | ||||||||||||||||||
|
Is there any news about this? I am trying to use a change stream and filter only for updates to certain fields like this: | ||||||||||||||||||
| Comment by Asya Kamsky [ 22/Apr/19 ] | ||||||||||||||||||
|
To allow easier searching, adding the term "updatedFields" to this ticket.
| ||||||||||||||||||
| Comment by Gil Abrantes [ 15/Aug/18 ] | ||||||||||||||||||
|
Thank you Asya, that was useful to me just now. Adding up a few details, the full watch command after defining the getValue function to would be:
All the getValue does is build the content of the $let operator on the client side.
Using pymongo it would become (thanks Wan Bachtiar at mongodb-user!):
| ||||||||||||||||||
| Comment by Asya Kamsky [ 01/Jun/18 ] | ||||||||||||||||||
|
In cases where another expression needs to be compared to dotted field (for instance in change streams), this workaround can be used:
| ||||||||||||||||||
| Comment by Asya Kamsky [ 01/Jun/18 ] | ||||||||||||||||||
|
As of 3.6 this can be done in find via $expr. Something like this in the shell:
| ||||||||||||||||||
| Comment by Asya Kamsky [ 08/Mar/16 ] | ||||||||||||||||||
|
Here's the aggregation query for it:
| ||||||||||||||||||
| Comment by Asya Kamsky [ 07/Mar/16 ] | ||||||||||||||||||
|
There are two workarounds to this, I put them in the SO question as an answer. Mentioning here in case someone finds this ticket before there's a fix and needs a workaround to query for such entries in the oplog. You can do it via aggregation or mapReduce. | ||||||||||||||||||
| Comment by Tanner Faulkner [ 11/Jan/16 ] | ||||||||||||||||||
|
Would also make it difficult to find updates to only property d in that example. Currently I can at least grep for it if I dump to file. | ||||||||||||||||||
| Comment by Eric Milkie [ 11/Jan/16 ] | ||||||||||||||||||
|
Right, but we can put the whole document in, for idempotency. It would mean the data stream could potentially get a lot heavier (undesirable). | ||||||||||||||||||
| Comment by Lee Parayno [ 11/Jan/16 ] | ||||||||||||||||||
|
I believe the first one would replace the whole document for "b", whereas the second one would only update the "b.d" key. Lee Parayno | ||||||||||||||||||
| Comment by Eric Milkie [ 11/Jan/16 ] | ||||||||||||||||||
|
Since the following seem to be equivalent (are they?):
...perhaps we can simply change the oplog output for the $set modifier. It shouldn't affect applied operations in replication. | ||||||||||||||||||
| Comment by Kelsey Schubert [ 15/Dec/15 ] | ||||||||||||||||||
|
Hi lparayno, Thank you for the report: I am sending this ticket to be scheduled. Kind regards,
|