Currently, the server parameter maxIndexBuildMemoryUsageMegabytes, used to limit memory usage for index builds, has a minimum of 50MB. This memory limit, however, is split evenly across all indexes that are build by a single createIndexes(..) invocation, and as such the memory limit for each index can be quite low if a user is building multiple indexes at once or during initial sync. Having a memory limit that is too low for an index build can have detrimental effects on the duration it takes to complete due to requiring intermediate merges of spill ranges to remain below the memory limit, thus having negative impact on availability.

Previously SERVER-106589 was implemented to increase the memory available to index builds during initial sync, but as it was observed in SERVER-111871 (internal analysis: Index Build Memory Limit Performance Analysis), simply increasing the amount of memory available to index builds can have the negative effect of increasing the length of time it takes to build each index in cases where the amount of data to index exceeds the memory limits and spilling to disk is required.

As such, we should implement a mechanism that avoids starving multi-index builds with memory limits that are too low, while avoiding the negative effects of setting the memory limit to be too high.