Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17907

B-tree eviction blocks access to collection for extended period under WiredTiger

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.0.3, 3.1.2
    • Component/s: WiredTiger
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Completed:

      Description

      The sweep server under some conditions will evict entire b-trees, and for a large b-tree this can take an extended time (many minutes in some cases). While this is occurring any attempt to access the b-tree hangs.

      Example: start mongod with --storageEngine wiredTiger --wiredTigerCacheSizeGB 9. Then populate a collection with 8 GB of collection and index data (using small documents as below probably exacerbates the issue because it makes the eviction slower due to the number of buffers that must be freed during the eviction):

      for t in $(seq 16); do
          mongo test --eval "         
              every = 10000
              for (var i=0; ; ) {
                  var bulk = db.c.initializeUnorderedBulkOp();
                  for (var j=0; j<every; j++, i++)
                      bulk.insert({})
                  bulk.execute();
                  print(i)
                  
                  s = db.c.stats(1024*1024)
                  if ($t==1)
                      print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
                  if (s.size+s.totalIndexSize >= 8000)
                      break
              }
          "
      done
      

      Start mongostat to monitor the cache statistics while paging in the b-trees using this command:

      mongo test --eval 'printjson(db.c.find({x:1}).hint({_id:1}).toArray())'
      

      While this is running you should observe the cache utilization climb to its limit, 80%.

      Now wait a minute or two (so that the b-trees haven't been accessed in a while), then trigger eviction of the test.c collection and _id index by creating a new test.d collection (20 MB will do):

      for t in $(seq 16); do
          mongo test --eval "         
              every = 10000
              for (var i=0; ; ) {
                  var bulk = db.d.initializeUnorderedBulkOp();
                  for (var j=0; j<every; j++, i++)
                      bulk.insert({})
                  bulk.execute();
                  print(i)
                  
                  s = db.d.stats(1024*1024)
                  if ($t==1)
                      print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
                  if (s.size+s.totalIndexSize >= 20)
                      break
              }
          "
      done
      

      Soon after this (within a minute) the sweep server will begin evicting the test.c collection and _id index b-trees; you will see cache utilization drop from 80% to 0% over the course of some time (a couple minutes, somewhat longer on Windows than Linux). During this time accesses to test.c block.

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: Don't sweep unless there's a reason we need to close file handles.
          Add a new configuration option, file_manager.open_handles that sets
          a minimum number of file handles that must be open before sweep runs,
          default is 250. Reference #1856, SERVER-17907.
          Branch: sweep-file-handle-minimum
          https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907 . Branch: sweep-file-handle-minimum https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: Don't sweep unless there's a reason we need to close file handles.
          Add a new configuration option, file_manager.open_handles that sets
          a minimum number of file handles that must be open before sweep runs,
          default is 250. Reference #1856, SERVER-17907.
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907 . Branch: develop https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

          Message: Don't sweep unless there's a reason we need to close file handles.
          Add a new configuration option, file_manager.open_handles that sets
          a minimum number of file handles that must be open before sweep runs,
          default is 250. Reference #1856, SERVER-17907.
          Branch: cursor-pin
          https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907 . Branch: cursor-pin https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

          Message: Discard trees from cache in the background.

          We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles). If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed.

          With this change, handles are marked "dead", and readers will no longer use them. The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on.

          refs SERVER-17907, SERVER-18014
          Branch: develop
          https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Discard trees from cache in the background. We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles). If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed. With this change, handles are marked "dead", and readers will no longer use them. The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on. refs SERVER-17907 , SERVER-18014 Branch: develop https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: