[SERVER-463] map/reduce should use all CPU cores Created: 08/Dec/09  Updated: 06/Dec/22  Resolved: 04/Feb/22

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Dwight Merriman Assignee: Backlog - Query Execution
Resolution: Done Votes: 157
Labels: map-reduce
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-9723 Enhance mapReduce to use all cores fo... Closed
Related
related to SERVER-13793 slow1/memory.js fails when building m... Closed
is related to SERVER-7760 Add a map/reduce test for V8 Closed
Assigned Teams:
Query Execution
Participants:
Case:

 Description   

instead of just 1

at least as an option



 Comments   
Comment by Esha Bhargava [ 04/Feb/22 ]

Closing these tickets as part of the deprecation of mapReduce.

Comment by Ben Becker [ 26/Feb/13 ]

Hi Antoine,

The goal of this ticket is to allow a single MapReduce job to use multiple cores, which is not yet implemented. This requires several new features; e.g. a storage system for intermediate results that doesn't acquire a db-level write lock, and global CPU/memory resource management.

Comment by Ben McCann [ 26/Feb/13 ]

I think there is agreement that M/R should use multiple cores. It's just a question of fixing it to do that now that we have v8.

Comment by Antoine Guiral [ 26/Feb/13 ]

Hum, so instead of 1 M/R at a time we could do "numCPU" M/R? I would prefer that my M/R become faster. Why they will still use only one CPU? When you perform a big M/R there is in fact a lot of M/R, they should be parallelized no? Or maybe I miss something..

Comment by Ben Becker [ 15/Feb/13 ]

To clarify a bit further, v2.4 will allow multiple MapReduce commands to execute JavaScript concurrently. A single MapReduce command will only use one core.

Comment by Tad Marshall [ 15/Feb/13 ]

The move to V8 does enable additional multi-threading options, but MapReduce itself needs some restructuring to take advantage of this properly. This won't be fully realized for 2.4, but it should probably be placed on the schedule and not left as "planned but not scheduled".

Comment by Ben McCann [ 15/Feb/13 ]

I'm surprised this is listed as "planned but not scheduled". Won't this be fixed as part of moving to v8 in 2.4?

Comment by Andrew Armstrong [ 11/Mar/11 ]

Would be good to have I think for sharded setups; given I think group() does not work when sharded for example either?

Comment by Russell Smith [ 28/Jul/10 ]

This is important to me, as it will significantly improve performance on my machine (I'm cpu bound on a single core, on a 8 core server).

Comment by Pen Fold [ 24/Jul/10 ]

Is it possible to schedule this improvement for an upcoming release?

This should give me a serious performance boost. Especially in a sharded setup.

Generated at Thu Feb 08 02:54:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.