Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19966

Excessive cursor caching in integration layer under WiredTiger

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Works as Designed
    • Affects Version/s: 3.0.5
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Labels:
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      With modest numbers of mongod client connections and collections a large number of WT cursors can be cached by the integration layer. Each cached WT cursor can consume significant memory (as much as the largest document it has handled), so this can result in very significant excess memory utilization and OOM. Repro:

      1000 connections:

      for t in $(seq 1000); do
          js test "load('test.js'); repro($t)" &
      done
      wait
      

      each connection updates 100 collections, each with 4 indexes:

      collections = 100
       
      function repro(t) {
          var round = 0;
          while (true) {
              round++;
              if (t==1)
                  print("=== Starting round: " + round)
              for (i=0; i<collections; i++) {
                  var c = db["col" + i]
                  if (round==1) {
                      c.ensureIndex({a:1})
                      c.ensureIndex({b:1})
                      c.ensureIndex({c:1})
                  }
                  c.insert({a:round, b:round, c:round})
              }
              if (t==1) {
                  cursors = db.serverStatus().wiredTiger.session['open cursor count']
                  print('cursors', cursors)
              }
          }
      }
      

      Result is more than 500k cached WT cursors:

      === Starting round: 1
      cursors 112025
      === Starting round: 2
      cursors 231014
      === Starting round: 3
      cursors 318091
      === Starting round: 4
      cursors 375483
      === Starting round: 5
      cursors 393430
      === Starting round: 6
      cursors 393760
      === Starting round: 7
      cursors 393950
      === Starting round: 8
      cursors 393950
      === Starting round: 9
      cursors 397336
      === Starting round: 10
      cursors 408085
      === Starting round: 11
      cursors 416445
      === Starting round: 12
      cursors 443923
      === Starting round: 13
      cursors 471652
      === Starting round: 14
      cursors 485585
      === Starting round: 15
      cursors 488255
      === Starting round: 16
      cursors 495570
      === Starting round: 17
      cursors 498425
      === Starting round: 18
      cursors 500035
      

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                4 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: