[SERVER-1050] not allowed to $push and $pop to same field in same update Created: 26/Apr/10  Updated: 06/Dec/22  Resolved: 29/Jun/19

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: 1.4.0
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Ian White Assignee: Backlog - Query Team (Inactive)
Resolution: Won't Do Votes: 167
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-27089 Extend the update subsystem to suppor... Closed
Related
related to SERVER-991 $push with $slice + $sort Closed
related to SERVER-6566 Support conditional updates: $updates Closed
is related to SERVER-2362 Add new Deque/Set operation modifiers Closed
is related to SERVER-2643 Allow Field Name Duplication with Mod... Closed
is related to SERVER-1947 can't push and set in one operation Closed
Assigned Teams:
Query
Participants:

 Description   

Was just talking about #991 with Dwight and noticed that you can't workaround by doing $push / $pop to the same field on the same update, for example:

> db.test.drop();
false
> db.test.save(

{ mylist: [1,2] }

);
> db.test.update( {}, { $push:

{ mylist: 3 }

, $pop:

{ mylist: -1 }

} );
Field name duplication not allowed with modifiers

Would be nice if this was permitted.



 Comments   
Comment by Asya Kamsky [ 30/Jun/19 ]

> Is this still atomic?

Yes, absolutely, like all updates on a single document, it is all-or-nothing, fully isolated/atomic.

Comment by NOVALUE Mitar [ 29/Jun/19 ]

Is this still atomic?

Comment by Asya Kamsky [ 29/Jun/19 ]

SERVER-40381 implemented support for aggregation expressions to specify update for 4.2. 

You can see some examples here.

This can be done by setting the array to its new value using various aggregation array expressions. I believe this can handle all of the examples/use cases that were mentioned in the comments.

The more complex "upsert to array" that was mentioned:

db.c.update({}, [
      {$set:{array:{$cond:{
            if: {$in:[ <newElement> ,  "$array.x"]},
            then:{$map:{ ... }} ,   // process array making appropriate change to correct element
            else: {$concatArrays: [ "$array", <newElement> ]}
      }}}}
]);

Comment by Asya Kamsky [ 09/May/18 ]

oleg@evergage.com you are correct that in 3.6 there are no workarounds for other use cases mentioned here.

If you are not talking about a sharded cluster, 4.0 will provide a workaround for this (by adding replica set transactions, which allows you to make several updates to the same document in the same transaction and receive ACID guarantees of making a single more complex update) but it won't be as efficient as making a single update would be.

We are currently planning work that would resolve this in the following major release, though of course no guarantees since that's a ways away still.

Comment by Oleg Rekutin [ 09/May/18 ]

These are great improvements in 3.6, Asya!

Unfortunately, we still cannot "upsert" an array element. Right now, it requires logic with an if-element-with-key-present update, falling back to an if-absent update (and then back to present again).

If I were able to execute $pull first, then I could remove any existing entries with the key and then re-add using a $push, effectively simulating an upsert.

Comment by Asya Kamsky [ 09/May/18 ]

Note that 3.6 added ability to update arrays with arrayFilters which handles several of the examples described in comments (not all):

to add results to multiple items in the array and it seems the best would be to pull off the original entries and then push the items with results added

 

 db.coll.update(
   {<match-condition>}, 
   {$set:{"array.$[e].newField":"abc"}},
   {arrayFilters:[
      {e:{otherField:"xxx"}}
   ]}
 )

 

This will use some match-condition to find documents to update, then update all array elements in "array" that have "otherField" equal to "xxx" by setting "newField" to "abc".

Comment by Matt Kalan [ 18/Oct/16 ]

Ran into needing this capability today working with a user. They want to add results to multiple items in the array and it seems the best would be to pull off the original entries and then push the items with results added

Comment by Benoit Labergri [ 20/Apr/16 ]

Quit similar requierment for us : conditional $pull (on sub document field in the array) and non conditional $push in the same update on the same array.

Comment by Mark Bigler [ 10/Oct/15 ]

+1

Comment by Ismet Ozalp [ 07/Aug/15 ]

This type of issues, always requires workarounds, most of the time these workarounds are ending up with ugly solutions.

Just like this one these are all 5 year issues.
SERVER-1243
SERVER-1920
Just like them i am going to watch this issue, hope for someday it will be fixed.

+1

Comment by Roman [ 16/Jul/15 ]

+1

Comment by Brett [ 07/Mar/15 ]

This would be very helpful when trying to move sub objects from one array to another.

Comment by rene mena [ 08/Sep/14 ]

+1

Comment by Neil S [ 22/Aug/14 ]

Also related to https://jira.mongodb.org/browse/SERVER-2643 I think

Comment by Eric Roberts [ 09/Jun/14 ]

Similar scenario as Kevin Rice. I need to upsert a variable amount of values into arrays but pad to a default max in a single call.

Comment by Thomas Boutell [ 16/May/14 ]

+1. Just hit this in the same scenario as Michael Dwan. When propagating permissions changes it makes sense to remove some and add some at the same time, and it's not a good idea to have competing threads doing it.

Comment by Kevin J. Rice [ 14/Feb/14 ]

I'm using an array to do padding for me. I want to start out with 'offsets' as an array of 100 elems, then pop off it each time I insert in 'vals'. this is fine, normally, I push vals and pop offsets. But, I want to do an upsert, and it apparently won't let me do setOnInsert of (offsets = array of 100 elems) AND pop from offsets (the normal case):

retval = self.colname.update(
    { 'xx' : 'aa' },
    { '$push' : {'vals' : datapoint}, 
       '$pop'  : {'offsets' : 1},
       '$setOnInsert' : { 'offsets' : offsets },
     }, upsert = True, multi = False, w = writeConcern )

I get the error:

OperationFailure: Field name duplication not allowed with modifiers

Comment by RobertWHurst [X] [ 07/Feb/14 ]

bump

Comment by RobertWHurst [X] [ 28/Nov/13 ]

This is a rather important issue. Is there any progress on this issue thus far? I'd like to remove the ugly hack we have in our orm to split diffs with both $pullAll and $push.

Comment by Debarshi Mukherjee [ 11/Jul/12 ]

My vote is there for this. This would be really useful to have. Trying to implement a kind of versioning strategy for mongo documents and though there may be other ways to bypass this current limitation, the end solution won't unfortunately be so tight.

Comment by Philipp Heinze [ 26/Mar/12 ]

Kekoa Vincent,

you may avoid the race Condition if you make the updates atomic by adding a temporary lock with the first update which is then removed with the second, so no two Threads may perform this operation for one document at the same time. Though you need some error handling for that in your app. Anyway this ticket needs to be resolved.

Comment by Kekoa Vincent [ 09/Mar/12 ]

Also agree that a general solution for this is needed, my use case is I need to refresh items in a list, so I $pull the items out of the list and $pushAll the new version of the items, leaving existing items that aren't being updated there unmodified. It would be great if this were atomic so there wouldn't be a race condition with other threads or processes doing the same thing and resulting in duplicates.

Order of operations does matter in this situation, so please consider that in the implementation. My workaround is to perform two updates, and cross my fingers that I don't have two threads running the updates on the same document.

Comment by Steve Green [ 05/Mar/12 ]

I agree with Jon D and also need the ability to $pull and $push from the same array in a single operation.

Comment by Andy Gayton [ 29/Dec/11 ]

For sure Scott, SERVER-991 would be ideal, if possible.

Comment by Scott Hernandez (Inactive) [ 29/Dec/11 ]

Andy, I've linked to SERVER-991 which is probably more what you want – a fixed size $push.

Comment by Andy Gayton [ 29/Dec/11 ]

Our use case is we keep versioned data as a list sub-document. Each time the data is changed the new version of the data is $push-ed on to the head of the list. To keep this list bounded, when the list reaches 60 versions, we $push the newest on to the head, and $pull the oldest 10 versions. This keeps the number of versions stable between 50-60. It'd be nice for this operation to be atomic.

Comment by Jon D [ 12/Oct/11 ]

I'd request that the ability to $pull and then $push in the same update operation be included as well. I have a scenario where I'm storing somewhat complex sub-documents inside an array where I want to $pull first on a query including equality and $lte operators (if any matching records exist), and then push a new version of the sub-document to the same array. Currently I need to issue two updates to accomplish this. This is a very common query in my system and will be issued millions of times daily, so this would be a nice little boost for me!

eg: { { $pull : { "a" : 2, { $lte :

{ "updated" : '2011-01-01' }

} } }, { $push :

{ "a" : 2, "b" : 3, "updated" : '2011-01-02' }

} }

Comment by Harald Lapp [ 23/May/11 ]

I think this should not only be limited to $push and $pop but to other atomic operations, too. For example: {$inc:

{'test': 0}

, $set: {'test': 50}}. While at first sight this might not make much sense, there is on the other hand no reason, why this should not work. I am doing some dynamic update operations on some collection, where the update object is built dynamically. I use "'$inc': 0" to have the possibility to initialize some object, if it has no value and leave it's value, if it's already there. The "$set" operation is added dynamically from the application (sometimes it's there, sometimes not), if required. I do not have to build complex logic structures in my script.

On the other hand: The database needs to now in which order the operation need to be performed. For example:

{'$inc':

{'test': 1}

, '$set': {'test': 50}} and {'$set':

{'test': 50}

, '$inc': {'test': 1}}

are totally different operations.

Comment by Michael Dwan [ 16/May/11 ]

+1 – We're trying to add and remove values from a permissions array field in a single query, however until this is resolved it requires 2 queries.

Comment by Charlie Yuan [ 16/Apr/11 ]

If this issue is resolved will it cover the use case from SERVER-2362?

Essentially, will it allow $pull and then $push in the same atomic operation to allow some sort of ordered set functionality where the array entries are distinct and ordered by last insertion as opposed to $addToSet where the entries are distinct but is ordered by initial insertion?

Comment by Mohammad Musa [ 16/Apr/11 ]

This is a basic array operation same as array[idx]="new value".

Why do we need to $pull and $addToSet?

It would be nice if we could use $elemMatch and then replace the value.

Comment by Leif Mortenson [ 04/Apr/11 ]

We ran into the need for this as well. I am able to work around it with multiple queries. But this will not be safe for some updates.

Comment by charso [ 29/Mar/11 ]

Has my vote.

Comment by drapeko [ 09/Dec/10 ]

Yeap, we definetely need this atomic operation

Comment by phpMoAdmin [ 16/Aug/10 ]

Having the same issue, I am trying to get stack-behavior via:
db.col.update(

{...}

, { $push: { stack:

{...val..}

}, $pop:

{ stack: -1 }

} );

Generated at Thu Feb 08 02:55:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.