-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.0.6, 2.1.1
-
Component/s: Index Maintenance, Internal Code
-
None
-
Environment:Affects all 2.0 and 2.1 releases, maybe 1.8.
-
ALL
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
The BSONObjExternalSorter performs in-memory sorts of sets of 1E6 BSON documents, writes the results to files, and then merges the files iteratively. Since a BSON document can be up to 16MB, and since BSONObjExternalSorter::add() stores a heap-allocated BSON object (rather than a datafile backed BSON object), an external sort can occupy up to 16 TB of heap space, leading to OOM killing.
Two possible fixes are (1) use the version of the BSON obj in the data files, and (2) limit the number of bytes we copy to the heap, instead of the number of objects.
Option 1 has the advantage of copy minimization, allowing us to sort more data in memory (because the sort representation consists of small objects) and reducing heap fragmentation.
Option 2 has the advantage of being a smaller source code change.
Before repairing, I would like to see a (standalone?) repro of this being a problem.