[SERVER-4644] aggregation: optimize memory utilitzation by selective reification of documents Created: 07/Jan/12 Updated: 27/Oct/15 Resolved: 12/Jul/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | 2.2.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Pasette (Inactive) | Assignee: | Mathias Stearn |
| Resolution: | Done | Votes: | 7 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
By analyzing the pipeline from back to front, if projections use inclusion, we can infer what fields we will need to evaluate the pipeline. If this is carried back to the beginning of the pipeline in a cumulative way (including indirect references in expressions), then we can get the complete set of fields that we need. Use this to limit the fields that are added to the reified documents at the head of the pipeline to reduce the amount of data passing through pipelines. May not be a significant change for pipelines that don't includes $sort or $group, but potentially a huge savings for those that do. |
| Comments |
| Comment by auto [ 20/Jul/12 ] |
|
Author: {u'date': u'2012-07-17T14:18:49-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: Rip out manageDependecies and DependencyTracker stuff |
| Comment by auto [ 20/Jul/12 ] |
|
Author: {u'date': u'2012-07-17T14:04:43-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: Hook up dependency detection to group, unwind, sort, skip, and limit |
| Comment by auto [ 20/Jul/12 ] |
|
Author: {u'date': u'2012-07-16T17:44:07-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: This commit lays the foundation for support across all projection operations |
| Comment by Mathias Stearn [ 12/Jul/12 ] |
|
Need to doc how to take advantage. |
| Comment by auto [ 12/Jul/12 ] |
|
Author: {u'date': u'2012-07-12T14:34:40-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: |
| Comment by auto [ 12/Jul/12 ] |
|
Author: {u'date': u'2012-06-29T16:49:56-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: If there is an early simple $project, apply it before converting to Documents This is a partial fix for This also provides a workaround for objects with types that aren't This commit will need some doc updates, in particular in the "Optimizing |
| Comment by auto [ 12/Jul/12 ] |
|
Author: {u'date': u'2012-06-29T16:49:56-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}Message: If there is an early simple $project, apply it before converting to Documents This is a partial fix for This also provides a workaround for objects with types that aren't This commit will need some doc updates, in particular in the "Optimizing |
| Comment by auto [ 15/Feb/12 ] |
|
Author: {u'login': u'cwestin', u'name': u'U-tellus\\cwestin', u'email': u'cwestin@10gen.com'}Message: checkpoint work for |
| Comment by auto [ 15/Feb/12 ] |
|
Author: {u'login': u'cwestin', u'name': u'U-tellus\\cwestin', u'email': u'cwestin@10gen.com'}Message: checkpoint dependency tracker for |
| Comment by Chris Westin [ 11/Jan/12 ] |
|
FREE-5540 would be solved by this, because we wouldn't be trying to add the binary fields to the reified documents – they're not involved in the aggregation in that case. |