[SERVER-82037] Memory used by sorter spills can grow without bound Created: 10/Oct/23 Updated: 14/Nov/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Gregory Noma | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Assigned Teams: |
Storage Execution
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
A sorter maintains a vector of "iterators" for each range of data that it spills to disk. Each allocation of this FileIterator is 128 bytes, plus the shared_ptr's control block. Thus with enough spills, the amount of memory being used can easily exceed the value configured by maxIndexBuildMemoryUsageMegabytes. This can also be exacerbated by building multiple indexes in a single index build, since the memory limit will be shared among the indexes and thus each one will spill more often. |
| Comments |
| Comment by Yuhong Zhang [ 10/Oct/23 ] |
|
Can be worked around by raising the value of maxIndexBuildMemoryUsageMegabytes when having lots of concurrent indexes being built (in each index build/createIndexes cmd). |