[SERVER-6296] improve aggregation framework concurrency implementation Created: 04/Jul/12  Updated: 28/Oct/15  Resolved: 22/Jul/13

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 2.5.2

Type: Improvement Priority: Major - P3
Reporter: Aaron Staple Assignee: Mathias Stearn
Resolution: Done Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-6123 aggregation holds read lock indefinit... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Revisit the concurrency options described in SERVER-6123.



 Comments   
Comment by Mathias Stearn [ 22/Jul/13 ]

Doc changes only needed if we already documented the locking behavior. If not, I think it makes sense to leave this as an undocumented internal implementation detail since there should be no behavior changes.

Comment by auto [ 22/Jul/13 ]

Author:

{u'username': u'RedBeard0531', u'name': u'Mathias Stearn', u'email': u'mathias@10gen.com'}

Message: SERVER-6296 Batch fetching in DocumentSourceCursor

The main win here is not grabbing and releasing the read lock for each
document to be processed.
Branch: master
https://github.com/mongodb/mongo/commit/8f0c10ec3f576b9c44213114ce8540f8a6698206

Comment by Mathias Stearn [ 22/Jul/13 ]

The implementation is the same as described above, however the batchsizes are always 4MB (MaxBytesToReturnToClientAtOnce) regardless of the number of documents or any yielding done while fetching them.

Comment by Mathias Stearn [ 21/Aug/12 ]

Current plan is to hold read lock only while in DocumentSourceCursor and have it fetch documents in batches of ~100 or until first yield point (such as doc not in memory). It can then release the lock and pass that batch of results down the pipeline. I think this is the best way to handle both multithreading the processing of the pipeline and the need to take a writelock for $out.

Generated at Thu Feb 08 03:11:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.