[SERVER-991] $push with $slice + $sort Created: 09/Apr/10  Updated: 06/Apr/23  Resolved: 13/Dec/12

Status: Closed
Project: Core Server
Component/s: Write Ops
Affects Version/s: None
Fix Version/s: 2.3.2

Type: New Feature Priority: Minor - P4
Reporter: Ian White Assignee: Alberto Lerner
Resolution: Done Votes: 141
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-6399 Refactor update() code Closed
is depended on by DOCS-932 Add documentation for fixed sized arrays Closed
Related
related to CSHARP-684 Add new features to Update builder fo... Closed
related to SERVER-8069 Support full $slice / $sort semantics... Closed
related to SERVER-8746 $push w/ $sort: make $slice optional Closed
is related to SERVER-8008 $push to sorted array Closed
is related to DOCS-874 Add New $push operator syntax Closed
is related to SERVER-1050 not allowed to $push and $pop to same... Closed
is related to SERVER-2363 $push/$pushAll support for insertion ... Closed
Participants:

 Description   

You can now limit the resulting elements in the array after you push. The new syntax is :

db.coll.update({...}, {$push: { $each:[<values>], $slice:-N, $sort:{...}}}

The initial implementation (see SERVER-8069 for more) only supports negative slice (taking stuff from the end of the array).

Sorting is by embedded document fields and doesn't work any type which is not an embedded docs (in the array).



 Comments   
Comment by Flavien [ 19/Jul/13 ]

Is there a chance to have this work with the aggregation framework in a $group operator, so that we can find the top K items in each group?

Comment by auto [ 03/Jan/13 ]

Author:

{u'date': u'2013-01-03T21:23:09Z', u'name': u'Alberto Lerner', u'email': u'alerner@10gen.com'}

Message: SERVER-991 SERVER-8008 Fixed test to use $slice instead of $trimTo.
Branch: master
https://github.com/mongodb/mongo/commit/56090cce56cc8e700b8dedc809e50523d3272cea

Comment by auto [ 03/Jan/13 ]

Author:

{u'date': u'2013-01-03T17:19:13Z', u'name': u'Alberto Lerner', u'email': u'alerner@10gen.com'}

Message: SERVER-991 SERVER-8008 Replaced $trimTo by $slice (but only accepting negative slices for now).
Branch: master
https://github.com/mongodb/mongo/commit/b3687e73545f6d0d04f1992119308e7ab80d44e1

Comment by auto [ 13/Dec/12 ]

Author:

{u'date': u'2012-12-13T18:48:38Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}

Message: SERVER-991 Fixed the last few trim's to trimTo's.
Branch: master
https://github.com/mongodb/mongo/commit/2beeb14b6a200094ccf1ebe3be638a8c0d00f843

Comment by auto [ 12/Dec/12 ]

Author:

{u'date': u'2012-12-12T21:21:43Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}

Message: SERVER-991 Allowed trimTo to be a double if it's not fractional.
Branch: master
https://github.com/mongodb/mongo/commit/7bf153e956d24bad98169ef751ff71050cce10dd

Comment by Geoffrey Gallaway [ 12/Dec/12 ]

Hmm, just tested this and it works great but I had to specifically cast the value for $trimTo to a NumberInt or NumberLong in the mongo shell. { $push : { 'v' :

{ $each : [ 19 ], $trimTo : 2 }

}} gave me "$trim value must be a numeric integer". Was that intended?

also, one thought.. Maybe the $trimTo values could be used as hints for padding factors? I would assume the use case for this involves updates to the same collections always having the same $trimTo values and starting off with only one value and growing until $trimTo when the values get popped off. This is probably over-optimization.

Comment by Alberto Lerner [ 12/Dec/12 ]

Added a new variation of $push that trims the resulting array to a given size. Sam and Alberto discussed already a bit about it.

Comment by Alberto Lerner [ 12/Dec/12 ]

Author:

{u'date': u'2012-12-12T15:01:10Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}

Message: SERVER-991 Fixed debug compilation.
Branch: master
https://github.com/mongodb/mongo/commit/90f4ef6e11b10732552ebf51d8c89d214b996ca9

Comment by auto [ 12/Dec/12 ]

Author:

{u'date': u'2012-12-12T14:56:18Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}

Message: SERVER-991 Added ability to trim array as part of a $push.
Branch: master
https://github.com/mongodb/mongo/commit/77cf08426e6e30c60efd7198c016386b52503a4a

Comment by Scott Lowe [ 05/Dec/12 ]

Definitely a much required basic feature - IMO this is more useful even than capped collections. I can't imagine it's that difficult to implement - is it?

Comment by Thomas [ 01/Aug/12 ]

This feature needs to exist. I'd almost consider just running a patched/forked version of mongodb just for this feature.

Comment by bishwa shrestha [ 29/May/12 ]

@Stephan B : +1 to that

Comment by Juliusz Gonera [ 29/May/12 ]

Why exactly has this been postponed again? What is the problem and how could the community help in resolving it?

Comment by Stephan B [ 29/May/12 ]

With $42M of new investment maybe this tiny 1-day-patch could be considered to be included in the 2.1/2.2 release? :-/

Comment by Jon Hoffman [ 24/May/12 ]

I implemented this with slightly different syntax than what's proposed. Haven't really tested out in production though:
https://github.com/hoffrocket/mongo/commits/jon-pushMax

Usage:

{ $pushMax : { a :

{max: 5, data: 4 }

} }

Also includes a new $hash (should probably be renamed $md5) query operation so that replication works correctly.

Comment by Richard Oliver [ 23/May/12 ]

@Stephan B absolutely, I've been waiting for nearly two years for this feature. It's really frustrating watching it slip further and further back.

Comment by Stephan B [ 23/May/12 ]

Is there ANY chance this might be re-considered for 2.2?

I mean it seems trivial to implement and is quite inefficient to solve in application code.
Based on the current roadmap the 2.4 release would probably be 2013 ...

I'd just like to understand why a small, but very popular feature like this can't make it into 2.1/2.2

Comment by Pierre Dane [ 01/May/12 ]

This feature is also really important for us. Would be great if this could get pushed up so it gets into 2.2. Thanks

Comment by Stephan B [ 17/Apr/12 ]

No! Why in 2.3? It says "Estimate: Small ( < 1 day)" - so why not in 2.2?
This feature request is 2+ years old already. I've been looking forward to this for months...

PS: According to "Popular Issues" this ranks #9.

Comment by Javier Ferrero [ 29/Mar/12 ]

+1 for this useful feature. Hope it gets resolved soon.

Comment by Eliot Horowitz (Inactive) [ 26/Mar/12 ]

Yes.

Comment by Colin Mollenhour [ 26/Mar/12 ]

Makes sense. However, that second to last push example is not valid JSON..

Did you mean this?

{ $push : { a : { $each: [ 1 ] , $trim : 5 } } }
{ $push : { a : { $each : [ 1 , 2 ] , $trim : 5 } } }

Comment by Eliot Horowitz (Inactive) [ 26/Mar/12 ]

We'd like to move more towards the way $addToSet works with $each

Current syntax:

{ $push : { a : 1 } }
{ $pushAll : { a : [ 1 , 2 ] } } 

newer

{ $push : { a : { $each : [ 1 , 2] } } }

given that, perhaps

{ $push : { a : { 1 , $trim : 5 } } }
{ $push : { a : { $each : [ 1 , 2 ] , $trim : 5 } } }

Comment by Victor Boivie [ 25/Mar/12 ]

@Colin: How will you handle a $pushAllTrim-syntax?. The argument for $pushAll is an array which would conflict with this proposal.

Comment by Colin Mollenhour [ 25/Mar/12 ]

In light of the discussion on github here is a new syntax proposal:

db.user.update({name: 'ibwhite'},{
  $pushTrim: {last_10_urls: ['/category/apple', -10]},
  $addToSetTrim: {last_5_unique_urls: ['/category/apple', -5]}
});

These operators take a 2-tuple as an argument for each key where the first value is the value to $push or $addToSet and the second is the length to trim to after the push. This holds with the "one operation per key" requirement while allowing different keys to have different trim lengths in one update (which the syntax in the description of this ticket does not allow).

Comment by Victor Boivie [ 25/Feb/12 ]

@Colin: I like that idea even though it breaks the rule that the same field can not be referenced by more than one modifier (SERVER-1050, which is rather tricky to solve in a generic way without rewriting too much code in update.cpp).

I took a stab at it (called it $trim however) and implemented support for it at https://github.com/mongodb/mongo/pull/181

Please review and/or comment it.

Comment by Colin Mollenhour [ 14/Feb/12 ]

Alternative syntax proposal which would not change any existing modifiers and would work with multiple capped arrays while allowing different sizes for each in one query and also allow for easy "ad-hoc" capping:

db.user.update({name: 'ibwhite'},{
  $push: {last_10_urls: '/category/apple'},
  $addToSet: {last_5_unique_urls: '/category/apple'},
  $prune: {
    last_10_urls: -10,  // negative numbers delete oldest elements
    last_5_unique_urls: -5,
  }
});

MongoDb would have to make the assumption that $prune always is applied after all other operators.

Comment by Auzzie Booth [ 31/Aug/11 ]

Please move this to 1.8.4 if possible. This feature is no too big but save a lot of queries and client-side processing in many scenarios

Comment by Martin Lazarov [ 03/Aug/11 ]

It's really useful and I hope it would be done in version 1.8.4

Comment by Paulo Lomanto [ 20/Mar/11 ]

+1 for this as this features brings a lot of new possibilities.

We have a similar situation as reported by Keith and it is very expensive to deal with a virtual capped array inside our application.

Comment by Keith Branton [ 18/Nov/10 ]

Dave, it is unclear if you are suggesting a mongo stored function (these tend to be a bit slow iir) or application server code.

I'm trying to track the last 100 (unique) members that view each profile in a social application. Collection is document per profile, and there is an array in that document called recentViewers.

Remember if you are going to do this in the application servers, i.e. load the profile document, change the array and store it again, with the few very popular pages the profiles have probably been updated several times since you loaded the document.

Not so easy, unless I'm missing something obvious (which is possible .

My biggest concern about this proposal is that the size is repeated at each point of use - not very dry. If updates are performed from more than one place in the code (maybe even implemented in different languages) then there is increased risk that the limits will not always be kept in sync, leading to unexpected results. I would much rather see a limit be stored in metadata in the database.

Perhaps a better way of implementing this functionality would be a new bson type; capped_array ::= int32 document, supporting all the usual array operations while imposing the limit

Comment by Dave Edelhart [ 18/Nov/10 ]

This may be a useful action but it doesn't follow the behavior of push as it is currently understood and implemented in most languages.

Because it implies a limit parameter, if implemented, it should be distinct and named_thus (push_stack(target, [elements], limit)? stack_add(target, [elements], limit)?

Again - this feels like a creeping attempt to force business logic into the model.

This can just as easily be accomplished with a few lines of javaScript or server side code.

Comment by Xiang.Song [ 08/Nov/10 ]

It's really useful. Hope it would be done by version 1.7 stable

Comment by Kevin Ryan (Inactive) [ 10/Oct/10 ]

any update or eta on this?
would love to have this!
thanks!

Comment by Suhail [ 06/Jun/10 ]

And $addToSet please =)

Comment by Karoly Negyesi [ 06/Jun/10 ]

Like a virtual capped collection, hm?

Generated at Thu Feb 08 02:55:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.