[SERVER-24153] Allow pipelines within $facet stage to process one batch at a time Created: 16/May/16  Updated: 19/Nov/16  Resolved: 12/Sep/16

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 3.3.14

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Charlie Swanson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Query 18 (08/05/16), Query 2016-08-29, Query 2016-09-19
Participants:

 Description   

As of SERVER-23654's resolution, the $facet stage currently puts everything into one giant buffer before passing results through the sub-pipelines. We should allow the sub-pipelines to process one batch at a time, removing the constraints of fitting the result set into the memory limit of 100MB.



 Comments   
Comment by Githook User [ 12/Sep/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-24153 Add unit tests for DocumentSource pausing behavior.
Branch: master
https://github.com/mongodb/mongo/commit/8383c8731ba18b8443e424783c5aa63a8aed9202

Comment by Githook User [ 07/Sep/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-24153 Add/refactor create() methods for DocumentSources

This will make it easier to add tests that each DocumentSource
correctly handles a paused input.
Branch: master
https://github.com/mongodb/mongo/commit/55ab9d613a96e7aa566fa01b5865ed871df0148b

Comment by Githook User [ 02/Sep/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-24153 Make DocumentSourceMock hold GetNextResults

This is prep work for adding tests that all stages can correctly handle
a kPauseExecution result.
Branch: master
https://github.com/mongodb/mongo/commit/0c6198abb7f17ba91c46ae73b4578580a3bb2b91

Comment by Githook User [ 01/Sep/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-24153 Split document_source_test.cpp into one file per stage.
Branch: master
https://github.com/mongodb/mongo/commit/698cd2555dabf2ab6c1ed4c504d1e2546da0f57a

Comment by Githook User [ 01/Sep/16 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-24153 Allow pipelines within $facet stage to process in batches.

This approach removes the need to buffer all documents in memory, thus
removing concerns about spilling intermediate results to disk.
Branch: master
https://github.com/mongodb/mongo/commit/b1014fe1b40a69cd90b27cb336a170317eecc6b7

Generated at Thu Feb 08 04:05:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.