Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18829

Cache usage exceeds configured maximum during index builds under WiredTiger

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:

      Description

      Issue Status as of Jul 14, 2015

      ISSUE SUMMARY
      MongoDB running with the WiredTiger storage engine may, during large index builds, exceed the amount of memory allocated to the database cache.

      This issue may prevent large index builds in some situations, such as during initial sync of new replica set members.

      USER IMPACT
      Excessive memory consumption may cause mongod to either abort with an out-of-memory condition, or be killed by the operating system's OOM killer, leading to a loss of availability of the affected node.

      WORKAROUNDS
      Lowering the amount of cache available to WiredTiger to the 1GB minimum may allow users affected by this issue to complete large index builds.

      AFFECTED VERSIONS
      MongoDB 3.0.0 through 3.0.4

      FIX VERSION
      The fix is included in the 3.0.5 production release.

      Original description

      This has been seen under somewhat different circumstances by a couple of customers.

      Initial sync of a large db (multi TB, couple billion documents). Foreground build of _id index starts at A, and cache usage rises steadily to 6x configured maximum before being terminated by OOM.

      Mongorestore of 100GB db. Multiple parallel background index builds begin at A; at B cache usage begins to grow until it reaches about 1.5x configured max and is terminated by OOM.

      No complete repro yet, although may have got a partial repro: during initial sync of a 500 GB db cache usage briefly rose to about 120% of configured max.

      1. ss-alex.html
        2.04 MB
        Bruce Lucas
      2. ss-alex.log
        6.06 MB
        Bruce Lucas
      1. incident.png
        183 kB
      2. oom.png
        73 kB
      3. partial-repro.png
        153 kB
      4. partial-repro-ckpt.png
        122 kB
      5. partial-repro-stacks.png
        178 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction.
          Also make sure that when eviction first needs to run, it can find some pages to
          evict.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction. Also make sure that when eviction first needs to run, it can find some pages to evict. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint

          SERVER-18829 Have pages start in the middle of the LRU queue for eviction
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint SERVER-18829 Have pages start in the middle of the LRU queue for eviction Branch: develop https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction.
          Also make sure that when eviction first needs to run, it can find some pages to
          evict.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction. Also make sure that when eviction first needs to run, it can find some pages to evict. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint

          SERVER-18829 Have pages start in the middle of the LRU queue for eviction
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint SERVER-18829 Have pages start in the middle of the LRU queue for eviction Branch: develop https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction.
          Also make sure that when eviction first needs to run, it can find some pages to
          evict.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction. Also make sure that when eviction first needs to run, it can find some pages to evict. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/32144696f78cf726b8b1df8becca0a86d870efa3
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint

          SERVER-18829 Have pages start in the middle of the LRU queue for eviction
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Merge pull request #2018 from wiredtiger/read-gen-midpoint SERVER-18829 Have pages start in the middle of the LRU queue for eviction Branch: develop https://github.com/wiredtiger/wiredtiger/commit/d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d
          Hide
          alexander.gorrod Alexander Gorrod added a comment -

          A fix for this is now in MongoDB master, and will be included in the 3.1.5 release.

          Show
          alexander.gorrod Alexander Gorrod added a comment - A fix for this is now in MongoDB master, and will be included in the 3.1.5 release.
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction.

          (cherry picked from commit d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d)
          Branch: mongodb-3.0
          https://github.com/wiredtiger/wiredtiger/commit/28c395baf4be3cdceb398fea80eb8f7b7513759c

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: SERVER-18829 Have pages start in the middle of the LRU queue for eviction. (cherry picked from commit d57dc26729bbc59c5bc3928aa90bb2ac3cd15d6d) Branch: mongodb-3.0 https://github.com/wiredtiger/wiredtiger/commit/28c395baf4be3cdceb398fea80eb8f7b7513759c

            People

            • Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: