Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-60331

External sorter can use more memory than the configured maximum

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.3.0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • ALL
    • Execution Team 2022-01-24, Execution Team 2022-02-07

      The external sorter, when it spills a sorted range, writes out chunks of 64KB. When it comes time to merge, it buffers at least one chunk in memory from each sorted range.

      The maxIndexBuildMemoryUsageMegabytes defaults to 200MB. By building many indexes in one createIndexes command, each build is given a fraction of that limit. This causes each index's sorter to spill more frequently than if each index were built separately.

      Since each spill costs up to 64K in memory during the merge phase, with 1000 spills, that's about 64MB for one index. If you're building 10 indexes at once, then you are looking at about 640MB of overhead, which is 3x the default limit.

            Assignee:
            jordi.olivares-provencio@mongodb.com Jordi Olivares Provencio
            Reporter:
            louis.williams@mongodb.com Louis Williams
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: