Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21585

Cache usage exceeds configured value, operations stall, WiredTigerLAS.wt grows

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 3.2.0-rc3
    • Fix Version/s: 3.2.0-rc4
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      Multiple tests are seeing a cluster of apparently related symptoms:

      • During the last portion of a checkpoint cache usage grows to well above the configured maximum
      • Operations stall, probably as a result of cache exceeding maximum
      • The WiredTigerLAS.wt file grows without bound, and circumstantial evidence related to the use of the WT lookaside cache connects this with the previous two symptoms.
      • FTDC data capture stalls

      Three examples from two different apps:

      1. update_repro.js
        1 kB
        Dan Pasette
      1. ex1.png
        188 kB
      2. ex2.png
        199 kB
      3. ex3.png
        195 kB
      4. oplog-stalls.png
        161 kB
      5. stalls.png
        175 kB
      6. update_repro.png
        154 kB

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'}

          Message: Import wiredtiger-wiredtiger-mongodb-3.2.0-rc3-199-g4d72349.tar.gz from wiredtiger branch mongodb-3.2

          ref: 4898aa4..4d72349

          2cf57a6 SERVER-21585 Don't use the lookaside file until the cache is stuck full.
          4ad8df7 SERVER-21585 Clean up LAS based on the current count of inserted records
          Branch: master
          https://github.com/mongodb/mongo/commit/e3cd63fcae3deb1140941a51c85564f098062a23

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'name': u'Ramon Fernandez', u'email': u'ramon@mongodb.com'} Message: Import wiredtiger-wiredtiger-mongodb-3.2.0-rc3-199-g4d72349.tar.gz from wiredtiger branch mongodb-3.2 ref: 4898aa4..4d72349 2cf57a6 SERVER-21585 Don't use the lookaside file until the cache is stuck full. 4ad8df7 SERVER-21585 Clean up LAS based on the current count of inserted records Branch: master https://github.com/mongodb/mongo/commit/e3cd63fcae3deb1140941a51c85564f098062a23
          Hide
          ramon.fernandez Ramon Fernandez added a comment -

          Using update_repro.js and Keith's changes the WiredTigerLAS.wt file has grown to 28MB after 1h and it's holding steady, so I've merged these changes into the mongo tree.

          Show
          ramon.fernandez Ramon Fernandez added a comment - Using update_repro.js and Keith's changes the WiredTigerLAS.wt file has grown to 28MB after 1h and it's holding steady, so I've merged these changes into the mongo tree.
          Hide
          bruce.lucas Bruce Lucas added a comment -

          Running with the patch build with an insert-only repro of the issue, it appears that the unbounded growth of the LAS file and of the cache are no longer occurring, although we are still seeing significant stalls of zero throughput associated with checkpoints, up to 30s in this test.

          I believe this is a regression relative to 3.0. Should we continue that on this ticket, or spin off a separate ticket?

          Show
          bruce.lucas Bruce Lucas added a comment - Running with the patch build with an insert-only repro of the issue, it appears that the unbounded growth of the LAS file and of the cache are no longer occurring, although we are still seeing significant stalls of zero throughput associated with checkpoints, up to 30s in this test. I believe this is a regression relative to 3.0. Should we continue that on this ticket, or spin off a separate ticket?
          Hide
          pasette Dan Pasette added a comment -

          Bruce, let's spin it off.

          Show
          pasette Dan Pasette added a comment - Bruce, let's spin it off.
          Hide
          bruce.lucas Bruce Lucas added a comment -

          Spun off SERVER-21652.

          Show
          bruce.lucas Bruce Lucas added a comment - Spun off SERVER-21652 .

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: