[SERVER-1050] not allowed to $push and $pop to same field in same update Created: 26/Apr/10 Updated: 06/Dec/22 Resolved: 29/Jun/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Write Ops |
| Affects Version/s: | 1.4.0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Ian White | Assignee: | Backlog - Query Team (Inactive) |
| Resolution: | Won't Do | Votes: | 167 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Assigned Teams: |
Query
|
||||||||||||||||||||||||||||||||
| Participants: |
Andy Gayton, Asya Kamsky, Backlog - Query Team, Benoit Labergri, Brett, Charlie Yuan, charso, Debarshi Mukherjee, drapeko, Eric Roberts, Harald Lapp, Ian White, Ismet Ozalp, Jon D, Kekoa Vincent, Kevin J. Rice, Leif Mortenson, Mark Bigler, Matt Kalan, Michael Dwan, Mohammad Musa, Neil S, NOVALUE Mitar, Oleg Rekutin, Philipp Heinze, phpMoAdmin, rene mena, RobertWHurst [X], Roman, Scott Hernandez, Steve Green, Thomas Boutell
|
||||||||||||||||||||||||||||||||
| Description |
|
Was just talking about #991 with Dwight and noticed that you can't workaround by doing $push / $pop to the same field on the same update, for example: > db.test.drop(); ); , $pop: { mylist: -1 } } ); Would be nice if this was permitted. |
| Comments |
| Comment by Asya Kamsky [ 30/Jun/19 ] | |||||||
|
> Is this still atomic? Yes, absolutely, like all updates on a single document, it is all-or-nothing, fully isolated/atomic. | |||||||
| Comment by NOVALUE Mitar [ 29/Jun/19 ] | |||||||
|
Is this still atomic? | |||||||
| Comment by Asya Kamsky [ 29/Jun/19 ] | |||||||
|
You can see some examples here. This can be done by setting the array to its new value using various aggregation array expressions. I believe this can handle all of the examples/use cases that were mentioned in the comments. The more complex "upsert to array" that was mentioned:
| |||||||
| Comment by Asya Kamsky [ 09/May/18 ] | |||||||
|
oleg@evergage.com you are correct that in 3.6 there are no workarounds for other use cases mentioned here. If you are not talking about a sharded cluster, 4.0 will provide a workaround for this (by adding replica set transactions, which allows you to make several updates to the same document in the same transaction and receive ACID guarantees of making a single more complex update) but it won't be as efficient as making a single update would be. We are currently planning work that would resolve this in the following major release, though of course no guarantees since that's a ways away still. | |||||||
| Comment by Oleg Rekutin [ 09/May/18 ] | |||||||
|
These are great improvements in 3.6, Asya! Unfortunately, we still cannot "upsert" an array element. Right now, it requires logic with an if-element-with-key-present update, falling back to an if-absent update (and then back to present again). If I were able to execute $pull first, then I could remove any existing entries with the key and then re-add using a $push, effectively simulating an upsert. | |||||||
| Comment by Asya Kamsky [ 09/May/18 ] | |||||||
|
Note that 3.6 added ability to update arrays with arrayFilters which handles several of the examples described in comments (not all):
This will use some match-condition to find documents to update, then update all array elements in "array" that have "otherField" equal to "xxx" by setting "newField" to "abc". | |||||||
| Comment by Matt Kalan [ 18/Oct/16 ] | |||||||
|
Ran into needing this capability today working with a user. They want to add results to multiple items in the array and it seems the best would be to pull off the original entries and then push the items with results added | |||||||
| Comment by Benoit Labergri [ 20/Apr/16 ] | |||||||
|
Quit similar requierment for us : conditional $pull (on sub document field in the array) and non conditional $push in the same update on the same array. | |||||||
| Comment by Mark Bigler [ 10/Oct/15 ] | |||||||
|
+1 | |||||||
| Comment by Ismet Ozalp [ 07/Aug/15 ] | |||||||
|
This type of issues, always requires workarounds, most of the time these workarounds are ending up with ugly solutions. Just like this one these are all 5 year issues. +1 | |||||||
| Comment by Roman [ 16/Jul/15 ] | |||||||
|
+1 | |||||||
| Comment by Brett [ 07/Mar/15 ] | |||||||
|
This would be very helpful when trying to move sub objects from one array to another. | |||||||
| Comment by rene mena [ 08/Sep/14 ] | |||||||
|
+1 | |||||||
| Comment by Neil S [ 22/Aug/14 ] | |||||||
|
Also related to https://jira.mongodb.org/browse/SERVER-2643 I think | |||||||
| Comment by Eric Roberts [ 09/Jun/14 ] | |||||||
|
Similar scenario as Kevin Rice. I need to upsert a variable amount of values into arrays but pad to a default max in a single call. | |||||||
| Comment by Thomas Boutell [ 16/May/14 ] | |||||||
|
+1. Just hit this in the same scenario as Michael Dwan. When propagating permissions changes it makes sense to remove some and add some at the same time, and it's not a good idea to have competing threads doing it. | |||||||
| Comment by Kevin J. Rice [ 14/Feb/14 ] | |||||||
|
I'm using an array to do padding for me. I want to start out with 'offsets' as an array of 100 elems, then pop off it each time I insert in 'vals'. this is fine, normally, I push vals and pop offsets. But, I want to do an upsert, and it apparently won't let me do setOnInsert of (offsets = array of 100 elems) AND pop from offsets (the normal case):
I get the error: OperationFailure: Field name duplication not allowed with modifiers | |||||||
| Comment by RobertWHurst [X] [ 07/Feb/14 ] | |||||||
|
bump | |||||||
| Comment by RobertWHurst [X] [ 28/Nov/13 ] | |||||||
|
This is a rather important issue. Is there any progress on this issue thus far? I'd like to remove the ugly hack we have in our orm to split diffs with both $pullAll and $push. | |||||||
| Comment by Debarshi Mukherjee [ 11/Jul/12 ] | |||||||
|
My vote is there for this. This would be really useful to have. Trying to implement a kind of versioning strategy for mongo documents and though there may be other ways to bypass this current limitation, the end solution won't unfortunately be so tight. | |||||||
| Comment by Philipp Heinze [ 26/Mar/12 ] | |||||||
|
Kekoa Vincent, you may avoid the race Condition if you make the updates atomic by adding a temporary lock with the first update which is then removed with the second, so no two Threads may perform this operation for one document at the same time. Though you need some error handling for that in your app. Anyway this ticket needs to be resolved. | |||||||
| Comment by Kekoa Vincent [ 09/Mar/12 ] | |||||||
|
Also agree that a general solution for this is needed, my use case is I need to refresh items in a list, so I $pull the items out of the list and $pushAll the new version of the items, leaving existing items that aren't being updated there unmodified. It would be great if this were atomic so there wouldn't be a race condition with other threads or processes doing the same thing and resulting in duplicates. Order of operations does matter in this situation, so please consider that in the implementation. My workaround is to perform two updates, and cross my fingers that I don't have two threads running the updates on the same document. | |||||||
| Comment by Steve Green [ 05/Mar/12 ] | |||||||
|
I agree with Jon D and also need the ability to $pull and $push from the same array in a single operation. | |||||||
| Comment by Andy Gayton [ 29/Dec/11 ] | |||||||
|
For sure Scott, | |||||||
| Comment by Scott Hernandez (Inactive) [ 29/Dec/11 ] | |||||||
|
Andy, I've linked to | |||||||
| Comment by Andy Gayton [ 29/Dec/11 ] | |||||||
|
Our use case is we keep versioned data as a list sub-document. Each time the data is changed the new version of the data is $push-ed on to the head of the list. To keep this list bounded, when the list reaches 60 versions, we $push the newest on to the head, and $pull the oldest 10 versions. This keeps the number of versions stable between 50-60. It'd be nice for this operation to be atomic. | |||||||
| Comment by Jon D [ 12/Oct/11 ] | |||||||
|
I'd request that the ability to $pull and then $push in the same update operation be included as well. I have a scenario where I'm storing somewhat complex sub-documents inside an array where I want to $pull first on a query including equality and $lte operators (if any matching records exist), and then push a new version of the sub-document to the same array. Currently I need to issue two updates to accomplish this. This is a very common query in my system and will be issued millions of times daily, so this would be a nice little boost for me! eg: { { $pull : { "a" : 2, { $lte : { "updated" : '2011-01-01' }} } }, { $push : { "a" : 2, "b" : 3, "updated" : '2011-01-02' }} } | |||||||
| Comment by Harald Lapp [ 23/May/11 ] | |||||||
|
I think this should not only be limited to $push and $pop but to other atomic operations, too. For example: {$inc: {'test': 0}, $set: {'test': 50}}. While at first sight this might not make much sense, there is on the other hand no reason, why this should not work. I am doing some dynamic update operations on some collection, where the update object is built dynamically. I use "'$inc': 0" to have the possibility to initialize some object, if it has no value and leave it's value, if it's already there. The "$set" operation is added dynamically from the application (sometimes it's there, sometimes not), if required. I do not have to build complex logic structures in my script. On the other hand: The database needs to now in which order the operation need to be performed. For example: {'$inc': {'test': 1}, '$set': {'test': 50}} and {'$set': {'test': 50}, '$inc': {'test': 1}} are totally different operations. | |||||||
| Comment by Michael Dwan [ 16/May/11 ] | |||||||
|
+1 – We're trying to add and remove values from a permissions array field in a single query, however until this is resolved it requires 2 queries. | |||||||
| Comment by Charlie Yuan [ 16/Apr/11 ] | |||||||
|
If this issue is resolved will it cover the use case from Essentially, will it allow $pull and then $push in the same atomic operation to allow some sort of ordered set functionality where the array entries are distinct and ordered by last insertion as opposed to $addToSet where the entries are distinct but is ordered by initial insertion? | |||||||
| Comment by Mohammad Musa [ 16/Apr/11 ] | |||||||
|
This is a basic array operation same as array[idx]="new value". Why do we need to $pull and $addToSet? It would be nice if we could use $elemMatch and then replace the value. | |||||||
| Comment by Leif Mortenson [ 04/Apr/11 ] | |||||||
|
We ran into the need for this as well. I am able to work around it with multiple queries. But this will not be safe for some updates. | |||||||
| Comment by charso [ 29/Mar/11 ] | |||||||
|
Has my vote. | |||||||
| Comment by drapeko [ 09/Dec/10 ] | |||||||
|
Yeap, we definetely need this atomic operation | |||||||
| Comment by phpMoAdmin [ 16/Aug/10 ] | |||||||
|
Having the same issue, I am trying to get stack-behavior via: , { $push: { stack: {...val..}}, $pop: { stack: -1 }} ); |