[SERVER-15726] Improve replication of $slice commands Created: 20/Oct/14 Updated: 26/Nov/18 Resolved: 26/Nov/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.6.5 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Nikolay Grebnev | Assignee: | Asya Kamsky |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
We use slice for users activity feed limitation at website.
I would like to ask to improve slice function, so, when it is used, traffic for replicas would be the same as without slice being used. |
| Comments |
| Comment by Asya Kamsky [ 26/Nov/18 ] |
|
As long as our replication relies on all operations being recorded in idempotent format, this cannot be changed.
|
| Comment by Asya Kamsky [ 26/Nov/18 ] |
|
The issue is you are using $slice with negative offset which means we keep the last N elements of the array after the $push. When that does not change the number of elements in the array, we detect it as a "noop" however, when this causes elements at the beginning of the array to be removed then each remaining element in the array changes position! So we have to record the new position of each element to make the operation idempotent, and therefore we cannot represent this operation in any other way.
|
| Comment by Ramon Fernandez Marina [ 18/Mar/15 ] |
|
ngrebnev, this is the current, expected behavior for $slice. I'm going to re-purpose this ticket as an improvement request as per your initial suggestion on the ticket. Regards, |
| Comment by Ramon Fernandez Marina [ 10/Jan/15 ] |
|
Hi ngrebnev, apologies for the long delay. I do see the behavior you describe, and I'm investigating whether it is expected or not, and if it is whether there's room for improvement. It's easy to see how a $push operation would only need to send the pushed elements to the secondaries, but a $slice operation may change an array significantly so it could be that the only way to replicate such operation is to send the whole array to the secondaries. This being said, having very large arrays in documents is often not a good idea, and if you're keeping logs of some kind you may want to investigate the use of capped collections. |
| Comment by Nikolay Grebnev [ 20/Oct/14 ] |
|
$mongo['feed_activity'].update( {'_id'=>1}, {'$push'=>{log_name=> {'$each'=>[info_hash],'$slice'=>-10000}}},{:upsert=>true}) |