[SERVER-82037] Memory used by sorter spills can grow without bound Created: 10/Oct/23  Updated: 14/Nov/23

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Gregory Noma Assignee: Backlog - Storage Execution Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-68125 Index build on multi-key fields can c... Closed
is related to SERVER-83145 Shared buffer fragment incorrectly tr... Closed
Assigned Teams:
Storage Execution
Operating System: ALL
Participants:
Case:

 Description   

A sorter maintains a vector of "iterators" for each range of data that it spills to disk. Each allocation of this FileIterator is 128 bytes, plus the shared_ptr's control block. Thus with enough spills, the amount of memory being used can easily exceed the value configured by maxIndexBuildMemoryUsageMegabytes. This can also be exacerbated by building multiple indexes in a single index build, since the memory limit will be shared among the indexes and thus each one will spill more often.



 Comments   
Comment by Yuhong Zhang [ 10/Oct/23 ]

Can be worked around by raising the value of maxIndexBuildMemoryUsageMegabytes when having lots of concurrent indexes being built (in each index build/createIndexes cmd).

Generated at Thu Feb 08 06:48:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.