[SERVER-25318] Limit total memory utilization for bulk index builds Created: 28/Jul/16  Updated: 10/May/18  Resolved: 07/Nov/16

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Storage
Affects Version/s: None
Fix Version/s: 3.2.13, 3.4.0-rc3

Type: Bug Priority: Major - P3
Reporter: Bruce Lucas (Inactive) Assignee: Eric Milkie
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Documented
is documented by DOCS-9454 Docs for SERVER-25318: Limit total me... Closed
Related
Backwards Compatibility: Minor Change
Operating System: ALL
Backport Requested:
v3.2
Sprint: Storage 2016-11-21
Participants:
Linked BF Score: 0

 Description   

A bulk index build of multiple indexes on a collection, for example using createIndexes, or during an initial sync, will use up to 100 MB of memory per index, or up to 6400 MB of memory total (given the limit of 64 indexes per collection). This can lead to out-of-memory on smaller machines. We should limit the total memory used on smaller machines by reducing the amount of memory used per index build, and/or limiting the number of simultaneous index builds.



 Comments   
Comment by Githook User [ 02/Feb/17 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-25318 limit total memory consumed by bulk simultaneous index builds

(cherry picked from commit b20282c5fec7cdedc9ba0d1aaf8439bfcd226753)

Conflicts:
src/mongo/db/catalog/index_create.cpp
src/mongo/db/index/index_access_method.cpp
Branch: v3.2
https://github.com/mongodb/mongo/commit/c15f5c7f7b9ccb5a921edb6b66dc790994cb5c65

Comment by Eric Milkie [ 07/Nov/16 ]

New server parameter: maxIndexBuildMemoryUsageMegabytes
Default value: 500

Note that this limits the amount of memory that simultaneous foreground index builds for one collection will temporarily consume. In 3.4, initial sync clones one collection at a time, so no initial sync index build phase should consume more memory than the limit.
However, if a user initiates foreground index builds for collections in multiple databases at the same time, each databases' index builds will consume up to the memory limit, per database. (It is not possible to build indexes in the foreground on multiple collections within the same database simultaneously.)

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'milkie', u'name': u'Eric Milkie', u'email': u'milkie@10gen.com'}

Message: SERVER-25318 limit total memory consumed by bulk simultaneous index builds
Branch: master
https://github.com/mongodb/mongo/commit/b20282c5fec7cdedc9ba0d1aaf8439bfcd226753

Generated at Thu Feb 08 04:08:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.