[SERVER-44984] Reduce index thread pool size and reduce memory used per build Created: 06/Dec/19 Updated: 29/Oct/23 Resolved: 04/Mar/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance |
| Affects Version/s: | 4.2.1 |
| Fix Version/s: | 4.2.3, 4.4.0-rc0, 4.7.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Eric Milkie | Assignee: | Eric Milkie |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | KP42 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2019-12-16, Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2020-02-10, Execution Team 2019-12-30, Execution Team 2020-02-24, Execution Team 2020-03-09 | ||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 16 | ||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
The index build thread pool size, which is used by index builds on secondary nodes in 4.2 and for index builds on all nodes in 4.4+, was reduced to 3. This means that for a set of indexes for a collection, built simultaneously via one createIndexes command, each of these is limited to 3 concurrently running on secondary nodes in 4.2 and on all nodes in 4.4. Any index build sets in excess of three will block until another index build set finishes. In addition, the default memory cap used to build an index set was reduced from 500MB to 200MB. This memory cap governs how often the external sorter spills to disk. Original Description: The thread pool size is currently set to 10, but with today's default setting of 500MB for each external sorter, this limit makes it possible to exceed most instances' free memory. If we reduced the limit to something like 3, that would increase stability, without sacrificing functionality. |
| Comments |
| Comment by Githook User [ 03/Mar/20 ] | |
|
Author: {'username': 'milkie', 'name': 'Eric Milkie', 'email': 'milkie@10gen.com'}Message: This reverts commit b48cb5d8e2a8ccab0f4be401cc2533e5c2b650ae. | |
| Comment by Githook User [ 11/Feb/20 ] | |
|
Author: {'name': 'William Schultz', 'username': 'will62794', 'email': 'william.schultz@mongodb.com'}Message: Revert " This reverts commit 0be749a16f4523b2a23729e0754a89ba5ea99644. | |
| Comment by William Schultz (Inactive) [ 11/Feb/20 ] | |
|
This commit seems to have broken a number of jstestfuzz suites when it turned on the new index building behavior for some of these suites. This invariant seems to be occurring frequently in the failures:
| |
| Comment by Githook User [ 10/Feb/20 ] | |
|
Author: {'username': 'milkie', 'name': 'Eric Milkie', 'email': 'milkie@10gen.com'}Message: In addition, this commit turns on two-phase index building for some remaining concurrency suites, as the suites do not pass in one-phase mode with the smaller thread pool size. | |
| Comment by Githook User [ 14/Jan/20 ] | |
|
Author: {'name': 'Eric Milkie', 'email': 'milkie@mongodb.com', 'username': 'milkie'}Message: | |
| Comment by Eric Milkie [ 13/Dec/19 ] | |
|
In addition, we should reduce the default amount of memory the external sorter consumes before spilling, for index builds. I think we could reduce this by half without a material effect on performance. |