[SERVER-2398] for inline mapreduce, all emitted objects are kept in RAM before the 1st reduce, potential high memory usage Created: 24/Jan/11 Updated: 12/Jul/16 Resolved: 26/Jan/11 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.7.5 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Antoine Girbal | Assignee: | Antoine Girbal |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
during map phase, checkSize() is called to do reduceInMemory and potentially dumpToInc. |
| Comments |
| Comment by Antoine Girbal [ 25/Jan/11 ] |
|
Here is a test that shows problem ) } Use an emit that always uses the same key: function () { return sum; }function (key, vals) { return sum; Then apply MR: Added a fix where data gets reduced every 50KB IF there are potential duplicate. ], , |