Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17907

B-tree eviction blocks access to collection for extended period under WiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.0.3, 3.1.2
    • Affects Version/s: 3.0.1
    • Component/s: WiredTiger
    • None
    • Fully Compatible
    • ALL

      The sweep server under some conditions will evict entire b-trees, and for a large b-tree this can take an extended time (many minutes in some cases). While this is occurring any attempt to access the b-tree hangs.

      Example: start mongod with --storageEngine wiredTiger --wiredTigerCacheSizeGB 9. Then populate a collection with 8 GB of collection and index data (using small documents as below probably exacerbates the issue because it makes the eviction slower due to the number of buffers that must be freed during the eviction):

      for t in $(seq 16); do
          mongo test --eval "         
              every = 10000
              for (var i=0; ; ) {
                  var bulk = db.c.initializeUnorderedBulkOp();
                  for (var j=0; j<every; j++, i++)
                      bulk.insert({})
                  bulk.execute();
                  print(i)
                  
                  s = db.c.stats(1024*1024)
                  if ($t==1)
                      print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
                  if (s.size+s.totalIndexSize >= 8000)
                      break
              }
          "
      done
      

      Start mongostat to monitor the cache statistics while paging in the b-trees using this command:

      mongo test --eval 'printjson(db.c.find({x:1}).hint({_id:1}).toArray())'
      

      While this is running you should observe the cache utilization climb to its limit, 80%.

      Now wait a minute or two (so that the b-trees haven't been accessed in a while), then trigger eviction of the test.c collection and _id index by creating a new test.d collection (20 MB will do):

      for t in $(seq 16); do
          mongo test --eval "         
              every = 10000
              for (var i=0; ; ) {
                  var bulk = db.d.initializeUnorderedBulkOp();
                  for (var j=0; j<every; j++, i++)
                      bulk.insert({})
                  bulk.execute();
                  print(i)
                  
                  s = db.d.stats(1024*1024)
                  if ($t==1)
                      print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
                  if (s.size+s.totalIndexSize >= 20)
                      break
              }
          "
      done
      

      Soon after this (within a minute) the sweep server will begin evicting the test.c collection and _id index b-trees; you will see cache utilization drop from 80% to 0% over the course of some time (a couple minutes, somewhat longer on Windows than Linux). During this time accesses to test.c block.

        1. evicting-windows.png
          508 kB
          Bruce Lucas

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated:
              Resolved: