[SERVER-44984] Reduce index thread pool size and reduce memory used per build Created: 06/Dec/19  Updated: 29/Oct/23  Resolved: 04/Mar/20

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 4.2.1
Fix Version/s: 4.2.3, 4.4.0-rc0, 4.7.0

Type: Improvement Priority: Major - P3
Reporter: Eric Milkie Assignee: Eric Milkie
Resolution: Fixed Votes: 0
Labels: KP42
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Documented
is documented by DOCS-13432 Investigate changes in SERVER-44984: ... Closed
Duplicate
is duplicated by SERVER-44896 two phase index builds hangs in flow ... Closed
Problem/Incident
Related
related to SERVER-45262 make IndexBuildsCoordinator thread po... Closed
is related to SERVER-45099 Serialize index builds on secondary m... Closed
is related to SERVER-47155 Limit the number of simultaneous inde... Closed
is related to SERVER-54791 Excessive file descriptor usage durin... Closed
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2019-12-16, Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2020-02-10, Execution Team 2019-12-30, Execution Team 2020-02-24, Execution Team 2020-03-09
Participants:
Linked BF Score: 16

 Description   

The index build thread pool size, which is used by index builds on secondary nodes in 4.2 and for index builds on all nodes in 4.4+, was reduced to 3. This means that for a set of indexes for a collection, built simultaneously via one createIndexes command, each of these is limited to 3 concurrently running on secondary nodes in 4.2 and on all nodes in 4.4. Any index build sets in excess of three will block until another index build set finishes.

In addition, the default memory cap used to build an index set was reduced from 500MB to 200MB. This memory cap governs how often the external sorter spills to disk.

Original Description: The thread pool size is currently set to 10, but with today's default setting of 500MB for each external sorter, this limit makes it possible to exceed most instances' free memory. If we reduced the limit to something like 3, that would increase stability, without sacrificing functionality.



 Comments   
Comment by Githook User [ 03/Mar/20 ]

Author:

{'username': 'milkie', 'name': 'Eric Milkie', 'email': 'milkie@10gen.com'}

Message: SERVER-44984 reduce index build thread pool default and index build external sort memory limit default

This reverts commit b48cb5d8e2a8ccab0f4be401cc2533e5c2b650ae.
Branch: master
https://github.com/mongodb/mongo/commit/5f3625880188f9ca76bacb02c541abc2bd2d0f96

Comment by Githook User [ 11/Feb/20 ]

Author:

{'name': 'William Schultz', 'username': 'will62794', 'email': 'william.schultz@mongodb.com'}

Message: Revert "SERVER-44984 reduce index build thread pool default and index build external sort memory limit default"

This reverts commit 0be749a16f4523b2a23729e0754a89ba5ea99644.
Branch: master
https://github.com/mongodb/mongo/commit/b48cb5d8e2a8ccab0f4be401cc2533e5c2b650ae

Comment by William Schultz (Inactive) [ 11/Feb/20 ]

This commit seems to have broken a number of jstestfuzz suites when it turned on the new index building behavior for some of these suites. This invariant seems to be occurring frequently in the failures:

[ReplicaSetFixture:job0:node2] 2020-02-10T23:58:20.250+0000 F  -        [BackgroundSync] Invariant failure getReplSetMemberInStandaloneMode(opCtx->getServiceContext()) src\mongo\db\catalog\index_catalog_impl.cpp 122

Comment by Githook User [ 10/Feb/20 ]

Author:

{'username': 'milkie', 'name': 'Eric Milkie', 'email': 'milkie@10gen.com'}

Message: SERVER-44984 reduce index build thread pool default and index build external sort memory limit default

In addition, this commit turns on two-phase index building for some remaining concurrency suites, as the suites do not pass in one-phase mode with the smaller thread pool size.
Branch: master
https://github.com/mongodb/mongo/commit/0be749a16f4523b2a23729e0754a89ba5ea99644

Comment by Githook User [ 14/Jan/20 ]

Author:

{'name': 'Eric Milkie', 'email': 'milkie@mongodb.com', 'username': 'milkie'}

Message: SERVER-44984 reduce index build thread pool default and index build external sort memory limit default
Branch: v4.2
https://github.com/mongodb/mongo/commit/97f928dcabb5d8ca144b381dacde95337b20e405

Comment by Eric Milkie [ 13/Dec/19 ]

In addition, we should reduce the default amount of memory the external sorter consumes before spilling, for index builds. I think we could reduce this by half without a material effect on performance.

Generated at Thu Feb 08 05:07:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.