[DOCS-9454] Docs for SERVER-25318: Limit total memory utilization for bulk index builds Created: 02/Dec/16  Updated: 11/Sep/18  Resolved: 25/Oct/17

Status: Closed
Project: Documentation
Component/s: manual
Affects Version/s: 3.4.0
Fix Version/s: 3.4.0

Type: Task Priority: Major - P3
Reporter: Emily Hall Assignee: Steve Renaker (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
documents SERVER-25318 Limit total memory utilization for bu... Closed
Related
is related to DOCS-11355 Clarify usage of maxIndexBuildMemoryU... Closed
Participants:
Days since reply: 6 years, 16 weeks ago

 Description   

Engineering Ticket Description:

A bulk index build of multiple indexes on a collection, for example using createIndexes, or during an initial sync, will use up to 100 MB of memory per index, or up to 6400 MB of memory total (given the limit of 64 indexes per collection). This can lead to out-of-memory on smaller machines. We should limit the total memory used on smaller machines by reducing the amount of memory used per index build, and/or limiting the number of simultaneous index builds.



 Comments   
Comment by Githook User [ 25/Oct/17 ]

Author:

{'email': 'steve.renaker@mongodb.com', 'name': 'Steve Renaker', 'username': 'steveren'}

Message: DOCS-9454: Limit total memory utilization for bulk index builds
Branch: v3.4
https://github.com/mongodb/docs/commit/afe04a6ec609697b56d5b179dd16d71dca9d7c99

Comment by Githook User [ 25/Oct/17 ]

Author:

{'email': 'steve.renaker@mongodb.com', 'name': 'Steve Renaker', 'username': 'steveren'}

Message: DOCS-9454: Limit total memory utilization for bulk index builds
Branch: master
https://github.com/mongodb/docs/commit/c1628470c94a02d8344baebc4e75fb126c1e80f9

Comment by Eric Milkie [ 19/Oct/17 ]

Yes please! That sounds good to me.

Comment by Steve Renaker (Inactive) [ 19/Oct/17 ]

milkie The parameter maxIndexBuildMemoryUsageMegabytes isn't listed on the server parameters docs page, and there's no ticket to add it. I'll add it there as part of this ticket unless anyone objects.

Comment by Eric Milkie [ 18/Oct/17 ]

We're starting to get more questions about this one, so it would be good to put this somewhere. We should definitely put this in the Limits and Thresholds page, and probably also on this page: https://docs.mongodb.com/manual/core/index-creation/

Comment by Daniel Pasette (Inactive) [ 02/Feb/17 ]

This was backported to 3.2.13
I'm not sure where exactly this should be documented...

This is the relevant documentation information from SERVER-25318:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
New server parameter: maxIndexBuildMemoryUsageMegabytes
Default value: 500
Previously, the limit was 100MB per index, so if there were 64 indexes being built, it could use 64 * 100MB. Leaving the limit higher may result in faster index builds.

Note that this limits the amount of memory that simultaneous foreground index builds for one collection will temporarily consume. In 3.4, initial sync clones one collection at a time, so no initial sync index build phase should consume more memory than the limit.
However, if a user initiates foreground index builds for collections in multiple databases at the same time, each databases' index builds will consume up to the memory limit, per database. (It is not possible to build indexes in the foreground on multiple collections within the same database simultaneously.)

Generated at Thu Feb 08 07:58:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.