Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44984

Reduce index thread pool size and reduce memory used per build

    XMLWordPrintableJSON

Details

    • Fully Compatible
    • Execution Team 2019-12-16, Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2020-02-10, Execution Team 2019-12-30, Execution Team 2020-02-24, Execution Team 2020-03-09
    • 16

    Description

      The index build thread pool size, which is used by index builds on secondary nodes in 4.2 and for index builds on all nodes in 4.4+, was reduced to 3. This means that for a set of indexes for a collection, built simultaneously via one createIndexes command, each of these is limited to 3 concurrently running on secondary nodes in 4.2 and on all nodes in 4.4. Any index build sets in excess of three will block until another index build set finishes.

      In addition, the default memory cap used to build an index set was reduced from 500MB to 200MB. This memory cap governs how often the external sorter spills to disk.

      Original Description: The thread pool size is currently set to 10, but with today's default setting of 500MB for each external sorter, this limit makes it possible to exceed most instances' free memory. If we reduced the limit to something like 3, that would increase stability, without sacrificing functionality.

      Attachments

        Issue Links

          is documented by

          Task - A task that needs to be done. DOCS-13432 Investigate changes in SERVER-44984: Reduce index thread pool size

          • Major - P3 - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SERVER-44896 two phase index builds hangs in flow control ticket test suites with low ticket limit

          • Major - P3 - Major loss of function.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-45099 Serialize index builds on secondary members in 4.2

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-47155 Limit the number of simultaneous index builds running from user connections to 3

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-54791 Excessive file descriptor usage during index build with external sort

          • Major - P3 - Major loss of function.
          • Closed
          related to

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-45262 make IndexBuildsCoordinator thread pool configurable via startup parameter

          • Major - P3 - Major loss of function.
          • Closed

          Activity

            People

              milkie@mongodb.com Eric Milkie
              milkie@mongodb.com Eric Milkie
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: