[SERVER-60331] External sorter can use more memory than the configured maximum Created: 29/Sep/21 Updated: 29/Oct/23 Resolved: 02/Feb/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 5.3.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Louis Williams | Assignee: | Jordi Olivares Provencio |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Sprint: | Execution Team 2022-01-24, Execution Team 2022-02-07 | ||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
The external sorter, when it spills a sorted range, writes out chunks of 64KB. When it comes time to merge, it buffers at least one chunk in memory from each sorted range. The maxIndexBuildMemoryUsageMegabytes defaults to 200MB. By building many indexes in one createIndexes command, each build is given a fraction of that limit. This causes each index's sorter to spill more frequently than if each index were built separately. Since each spill costs up to 64K in memory during the merge phase, with 1000 spills, that's about 64MB for one index. If you're building 10 indexes at once, then you are looking at about 640MB of overhead, which is 3x the default limit. |
| Comments |
| Comment by Githook User [ 02/Feb/22 ] |
|
Author: {'name': 'Jordi Olivares Provencio', 'email': 'jordi.olivares-provencio@mongodb.com', 'username': 'jordiolivares'}Message: |