Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37926

Excess allocated memory associated with cursors due to WT "modify" operations

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.3
    • Component/s: Storage
    • Labels:
    • ALL
    • Storage Engines 2018-11-19
    • 1

      In SERVER-37795 we observed excess allocated memory associated with cursors due to WT "modify" operations in 3.6.8. This should not occur in 4.0 because improved cursor caching should limit the number of cursors, but we still see excess allocated memory:

      The excess memory in 4.0.3 is more than double that of 3.6.8. Note that the rate of updates is somewhat higher in 4.0.3 than 3.6.8 (probably SERVER-36221), but that doesn't appear to be enough larger to account for the >2x larger excess memory allocated in 4.0.3.

      Possibly related: "cached cursor count" seems to grow indefinitely, although we only have that information for the oplog collection and not globally.

      Notes:

      • Verified that the memory is associated with cursors by disabling cursor caching in 3.6.8 and the excess memory disappeared.
      • Uesed the heap profiler to confirmed that the excess allocated memory is coming from the same allocation sites in 4.0.3 as 3.6.8.

      Repro:

      load("/home/bdlucas/mongodb/git/mongo/jstests/libs/parallelTester.js")
      
      size = 1000000
      bunch = 100
      
      // start a bunch of threads inserting to a bunch of collections
      // in 3.6.8 this creates a big bunch of cursors (bunch^2, maybe)
      // but not in 4.0.3
      threads = []
      for (var t=0; t<bunch; t++) {
          thread = new ScopedThread(function(t) {
              c = db['c' + t]
              while (true)
                  c.insert({})
          }, t)
          threads.push(thread)
          thread.start()
      }
      
      // while that's going on do some small updates to large documents
      // this uses WT "modify" triggering the issue in SERVER-37795 in 3.6.8
      // that shouldn't happen in 4.0.3 but it does
      for (i = 0; i < 100; i++) {
          c = db['c' + i]
          x = 'x'.repeat(size)
          c.insert({_id: 0, x: x})
      }
      while (true) {
          for (i = 0; i < 100; i++) {
              c = db['c' + i]
              c.update({_id: 0}, {$inc: {i: 1}})
              c.find({_id: 0}).itcount()
          }
      }
      

        1. comparison.png
          comparison.png
          129 kB
        2. Screen Shot After WT-4418.png
          Screen Shot After WT-4418.png
          35 kB
        3. Screen Shot Before WT-4418.png
          Screen Shot Before WT-4418.png
          31 kB

            Assignee:
            vamsi.krishna@mongodb.com Vamsi Boyapati
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: