[SERVER-4644] aggregation: optimize memory utilitzation by selective reification of documents Created: 07/Jan/12  Updated: 27/Oct/15  Resolved: 12/Jul/12

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 2.2.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Daniel Pasette (Inactive) Assignee: Mathias Stearn
Resolution: Done Votes: 7
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-4930 aggregation: use dependency informat... Closed
is depended on by SERVER-5090 aggregation: use covered index Closed
is depended on by SERVER-5224 support index-only queries under aggr... Closed
Related
related to SERVER-4608 aggregation: allow binary data to pa... Closed
related to SERVER-5718 Code and CodeWScope should be able to... Closed
related to DOCS-1281 agg optimization when include project... Closed
is related to SERVER-447 new aggregation framework Closed
is related to SERVER-4445 aggregation: use a string table to r... Closed
is related to SERVER-4968 Aggregation fails with ambiguous asse... Closed
Participants:

 Description   

By analyzing the pipeline from back to front, if projections use inclusion, we can infer what fields we will need to evaluate the pipeline. If this is carried back to the beginning of the pipeline in a cumulative way (including indirect references in expressions), then we can get the complete set of fields that we need. Use this to limit the fields that are added to the reified documents at the head of the pipeline to reduce the amount of data passing through pipelines. May not be a significant change for pipelines that don't includes $sort or $group, but potentially a huge savings for those that do.



 Comments   
Comment by auto [ 20/Jul/12 ]

Author:

{u'date': u'2012-07-17T14:18:49-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: Rip out manageDependecies and DependencyTracker stuff SERVER-4644
Branch: master
https://github.com/mongodb/mongo/commit/94a0643a83b063678907c767a62066679f618981

Comment by auto [ 20/Jul/12 ]

Author:

{u'date': u'2012-07-17T14:04:43-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: Hook up dependency detection to group, unwind, sort, skip, and limit SERVER-4644
Branch: master
https://github.com/mongodb/mongo/commit/3e81903a1c824560c3cb77f7b6d6d6cd24860d74

Comment by auto [ 20/Jul/12 ]

Author:

{u'date': u'2012-07-16T17:44:07-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: SERVER-4644 Pull dependencies out of $project and use them in DocumentSourceCursor

This commit lays the foundation for support across all projection operations
Branch: master
https://github.com/mongodb/mongo/commit/6226f70bf6967d87885d2da4f6129404ab238ce7

Comment by Mathias Stearn [ 12/Jul/12 ]

Need to doc how to take advantage.

Comment by auto [ 12/Jul/12 ]

Author:

{u'date': u'2012-07-12T14:34:40-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: SERVER-4644 In debug builds ensure that skipping a $project doesn't effect output
Branch: master
https://github.com/mongodb/mongo/commit/24e2a2c5f85a95bbc862630f5c4d16cf4419a1ef

Comment by auto [ 12/Jul/12 ]

Author:

{u'date': u'2012-06-29T16:49:56-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: If there is an early simple $project, apply it before converting to Documents SERVER-4644

This is a partial fix for SERVER-4644 in that it only works with an
explicit $project and only if that project is supported by the existing
Projection class used to implement the second argument to find().

This also provides a workaround for objects with types that aren't
supported by the Value class (SERVER-4608, SERVER-5718, and SERVER-4968.
Previous behavior was to assert with no possible workaround.

This commit will need some doc updates, in particular in the "Optimizing
Performance" section.
Branch: master
https://github.com/mongodb/mongo/commit/c62b02c1dbc95d0ed1a57231298aa2d81dd84c39

Comment by auto [ 12/Jul/12 ]

Author:

{u'date': u'2012-06-29T16:49:56-07:00', u'email': u'mathias@10gen.com', u'name': u'Mathias Stearn'}

Message: If there is an early simple $project, apply it before converting to Documents SERVER-4644

This is a partial fix for SERVER-4644 in that it only works with an
explicit $project and only if that project is supported by the existing
Projection class used to implement the second argument to find().

This also provides a workaround for objects with types that aren't
supported by the Value class (SERVER-4608, SERVER-5718, and SERVER-4968.
Previous behavior was to assert with no possible workaround.

This commit will need some doc updates, in particular in the "Optimizing
Performance" section.
Branch: master
https://github.com/mongodb/mongo/commit/c62b02c1dbc95d0ed1a57231298aa2d81dd84c39

Comment by auto [ 15/Feb/12 ]

Author:

{u'login': u'cwestin', u'name': u'U-tellus\\cwestin', u'email': u'cwestin@10gen.com'}

Message: checkpoint work for SERVER-4644
Branch: master
https://github.com/mongodb/mongo/commit/c4bf7b30c13a5175752f9296dd8a2bbfad6a8242

Comment by auto [ 15/Feb/12 ]

Author:

{u'login': u'cwestin', u'name': u'U-tellus\\cwestin', u'email': u'cwestin@10gen.com'}

Message: checkpoint dependency tracker for SERVER-4644
Branch: master
https://github.com/mongodb/mongo/commit/57ea070eb7c129935d9d96e7655dfd4467eef857

Comment by Chris Westin [ 11/Jan/12 ]

FREE-5540 would be solved by this, because we wouldn't be trying to add the binary fields to the reified documents – they're not involved in the aggregation in that case.

Generated at Thu Feb 08 03:06:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.