Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6161

BSONObjExternalSorter can consume 16TB of heap space

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 2.0.6, 2.1.1
    • None
    • Affects all 2.0 and 2.1 releases, maybe 1.8.
    • ALL

    Description

      The BSONObjExternalSorter performs in-memory sorts of sets of 1E6 BSON documents, writes the results to files, and then merges the files iteratively. Since a BSON document can be up to 16MB, and since BSONObjExternalSorter::add() stores a heap-allocated BSON object (rather than a datafile backed BSON object), an external sort can occupy up to 16 TB of heap space, leading to OOM killing.

      Two possible fixes are (1) use the version of the BSON obj in the data files, and (2) limit the number of bytes we copy to the heap, instead of the number of objects.

      Option 1 has the advantage of copy minimization, allowing us to sort more data in memory (because the sort representation consists of small objects) and reducing heap fragmentation.

      Option 2 has the advantage of being a smaller source code change.

      Before repairing, I would like to see a (standalone?) repro of this being a problem.

      Attachments

        Activity

          People

            schwerin@mongodb.com Andy Schwerin
            schwerin@mongodb.com Andy Schwerin
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: