[SERVER-47155] Limit the number of simultaneous index builds running from user connections to 3 Created: 27/Mar/20  Updated: 29/Oct/23  Resolved: 02/Apr/20

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: None
Fix Version/s: 4.4.0-rc0, 4.7.0

Type: Improvement Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Gregory Wlodarek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-46659 Make initial sync work with two phase... Closed
Documented
is documented by DOCS-13797 [SERVER] Limit the number of simultan... Closed
Related
related to SERVER-48617 Concurrent 'createIndexes' can block ... Closed
related to SERVER-59122 repeated blocked index build messages... Closed
related to SERVER-49215 Differentiate internal idents used fo... Closed
related to SERVER-44984 Reduce index thread pool size and red... Closed
related to SERVER-49948 [4.2] Introduce maxNumActiveUserIndex... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: Execution Team 2020-04-06
Participants:
Linked BF Score: 18

 Description   

Each index build (i.e. a single createIndexes command, which may build more than one index) is allowed to use up to 200MB of memory outside of the WT cache by default. This is controlled by maxIndexBuildMemoryUsageMegabytes. In an attempt to bound memory usage and reduce WT cache pressure, we introduced a limit the number of concurrent index builds started in SERVER-44984. This limit was bug-prone and also applied to index builds on secondaries. This ticket imposes a concurrent limit on index builds started by users on primaries, effectively the same observable behavior.

On primaries, any index builds started by a user over the limit will block until the number of concurrent index builds drops below the limit. The log message "Too many index builds running simultaneously, waiting until the number of active index builds is below the threshold" (ID 4715500) will be logged when the limit is reached and an index build has blocked.

This default limit can be raised with the maxNumActiveUserIndexBuilds setParameter. This can be changed at startup or runtime.

Original Description

Both the primary and secondary nodes will have an unlimited thread pool size. This is done to allow secondary nodes to startup as many index builders as necessary in order to prevent scheduling deadlocks during initial sync or oplog application.

When commands are run from user connections that need to create indexes, those commands will hang until there are less than 3 running index builder threads, or until the operation is interrupted.

 



 Comments   
Comment by Mark Callaghan (Inactive) [ 05/Aug/20 ]

Yes, you do. Closed mine as a dup. Sorry for the spam.

Comment by Eric Milkie [ 05/Aug/20 ]

We also have DOCS-13797 listed above as a linked ticket.

Comment by Mark Callaghan (Inactive) [ 05/Aug/20 ]

Filed DOCS-13809 for this to be documented for 4.4

Comment by Daniel Pasette (Inactive) [ 28/Jul/20 ]

louis.williams can you clarify that this is max number of concurrent createIndex commands that can be executed vs. the maximum number of index specifications that can be built concurrently, since the createIndex command can build multiple indexes concurrently. 

Comment by Mark Callaghan (Inactive) [ 28/Jul/20 ]

Will there be docs for this? I don't see any today via:
https://docs.mongodb.com/master/reference/operator/aggregation/lookup/?searchProperty=current&query=maxNumActiveUserIndexBuilds

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 Limit the number of simultaneous index builds running from user connections to 3

(cherry picked from commit c7818dde1a910b6828f24e1e9670688bdd5e5a68)
Branch: v4.4
https://github.com/mongodb/mongo/commit/8180323509e77006765bde544c8db0c3d8127317

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 IndexBuildsCoordinatorMongod unit tests should use the default thread pool options

(cherry picked from commit b3d8a4c33e27f8d4536e6526d2ebb44d289b89c4)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e6bf2e9f1e14913fdf5963d085a496328d0a52ca

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 MultiIndexBlock::init() should invariant the collection is held with an exclusive lock

(cherry picked from commit 2f23989e4e0f51e1248d978fa8d8229102ac3019)
Branch: v4.4
https://github.com/mongodb/mongo/commit/003a965a09beeb1e84b61094095cdd13ba738e32

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 Limit the number of simultaneous index builds running from user connections to 3
Branch: master
https://github.com/mongodb/mongo/commit/c7818dde1a910b6828f24e1e9670688bdd5e5a68

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 IndexBuildsCoordinatorMongod unit tests should use the default thread pool options
Branch: master
https://github.com/mongodb/mongo/commit/b3d8a4c33e27f8d4536e6526d2ebb44d289b89c4

Comment by Githook User [ 02/Apr/20 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-47155 MultiIndexBlock::init() should invariant the collection is held with an exclusive lock
Branch: master
https://github.com/mongodb/mongo/commit/2f23989e4e0f51e1248d978fa8d8229102ac3019

Generated at Thu Feb 08 05:13:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.