[SERVER-60331] External sorter can use more memory than the configured maximum Created: 29/Sep/21  Updated: 29/Oct/23  Resolved: 02/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.3.0

Type: Bug Priority: Major - P3
Reporter: Louis Williams Assignee: Jordi Olivares Provencio
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2022-01-24, Execution Team 2022-02-07
Participants:
Case:

 Description   

The external sorter, when it spills a sorted range, writes out chunks of 64KB. When it comes time to merge, it buffers at least one chunk in memory from each sorted range.

The maxIndexBuildMemoryUsageMegabytes defaults to 200MB. By building many indexes in one createIndexes command, each build is given a fraction of that limit. This causes each index's sorter to spill more frequently than if each index were built separately.

Since each spill costs up to 64K in memory during the merge phase, with 1000 spills, that's about 64MB for one index. If you're building 10 indexes at once, then you are looking at about 640MB of overhead, which is 3x the default limit.



 Comments   
Comment by Githook User [ 02/Feb/22 ]

Author:

{'name': 'Jordi Olivares Provencio', 'email': 'jordi.olivares-provencio@mongodb.com', 'username': 'jordiolivares'}

Message: SERVER-60331 Make ExternalSorter respect memory limits
Branch: master
https://github.com/mongodb/mongo/commit/d6a545506f7842ec6bc369ed7aedcdb8c2d40cd6

Generated at Thu Feb 08 05:49:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.