[SERVER-3167] during final reduce in sharded environment, all results are loaded into an in memory map Created: 30/May/11  Updated: 12/Jul/16  Resolved: 14/Jun/11

Status: Closed
Project: Core Server
Component/s: MapReduce
Affects Version/s: None
Fix Version/s: 1.9.1

Type: Bug Priority: Major - P3
Reporter: Antoine Girbal Assignee: Antoine Girbal
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

the target mongod does parallelsort from all servers, getting the records sorted.
It should just reduce and insert to temp collection.
Instead the result is emitted, which puts it into a map.
Then when all records are done, map is dumped to disk.
May be very costly in cpu and mem for large jobs



 Comments   
Comment by Antoine Girbal [ 14/Jun/11 ]

Some comments on fix:

  • records with similar key are put in a list, then reduced, then inserted in temp col
  • we may want to do extra reduce if the size of list gets too big (same as for sharded output)
  • it now takes 1 write lock for each insert instead of 1 lock for full dump
Generated at Thu Feb 08 03:02:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.