-
Type: Improvement
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 2.6.0-rc1
-
Component/s: Aggregation Framework
-
Major Change
-
Query 2018-07-30, Query 2018-08-13, Query 2018-08-27, Query 2018-09-10
-
(copied to CRM)
Using 2.6.0rc1 (Linux x86-64) I've been doing some research into speeding up some Aggregation use cases via Parallelisation. For the full investigation see here: http://pauldone.blogspot.co.uk/2014/03/mongoparallelaggregation.html
One of the main outcomes, was although a good speed-up can be achieved with multiple threads each running aggregate() on a subset of the collection's data, the main thing holding back further performance improvement was the threads queueing to write out to result collections in the same database, queueing for the DB write-lock.
In the tests, the $out operator http://docs.mongodb.org/master/reference/operator/aggregation/out/ is being used to specify different output collection for each thread's aggregate() invocation. However the $out operator does not allow one to specify a named database, in addition to a named collection. As a result, the same database as the aggregation's source collection is assumed and it's not possible to use different databases, to remove the write-lock bottleneck for such use cases.
Please consider enhancing the $out operator to support declaring a target database in addition to a target collection, in a similar manner to how this can already be achieved today in MongoDB's MapReduce function (specifically the mapReduce() function's 'out' option - http://docs.mongodb.org/manual/reference/method/db.collection.mapReduce/#mapreduce-out-mtd )
Thanks Paul
- depends on
-
SERVER-35895 Add ability for $out to write to remote hosts
- Closed
- is depended on by
-
SERVER-36081 Write auth tests for $out and bypassDocumentValidation
- Closed
-
SERVER-36832 Allow $out to different database
- Closed
- is duplicated by
-
SERVER-13547 Aggregation framework should support output to a different database
- Closed
-
SERVER-35898 Support writes to other databases using $out
- Closed
- related to
-
SERVER-51886 $lookup + $merge pipeline may fail to resolve views correctly when collection names collide
- Closed