[SERVER-17907] B-tree eviction blocks access to collection for extended period under WiredTiger Created: 07/Apr/15  Updated: 12/Oct/15  Resolved: 27/Apr/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.1
Fix Version/s: 3.0.3, 3.1.2

Type: Bug Priority: Critical - P2
Reporter: Bruce Lucas (Inactive) Assignee: Michael Cahill (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File evicting-windows.png    
Issue Links:
Depends
Related
related to SERVER-17157 Seeing pauses in YCSB performance wor... Closed
related to SERVER-18014 Dropping a collection can block creat... Closed
related to SERVER-17514 Don't close handles until a threshold... Closed
related to SERVER-18286 Adjust the value of the WiredTiger fi... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Participants:

 Description   

The sweep server under some conditions will evict entire b-trees, and for a large b-tree this can take an extended time (many minutes in some cases). While this is occurring any attempt to access the b-tree hangs.

Example: start mongod with --storageEngine wiredTiger --wiredTigerCacheSizeGB 9. Then populate a collection with 8 GB of collection and index data (using small documents as below probably exacerbates the issue because it makes the eviction slower due to the number of buffers that must be freed during the eviction):

for t in $(seq 16); do
    mongo test --eval "         
        every = 10000
        for (var i=0; ; ) {
            var bulk = db.c.initializeUnorderedBulkOp();
            for (var j=0; j<every; j++, i++)
                bulk.insert({})
            bulk.execute();
            print(i)
            
            s = db.c.stats(1024*1024)
            if ($t==1)
                print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
            if (s.size+s.totalIndexSize >= 8000)
                break
        }
    "
done

Start mongostat to monitor the cache statistics while paging in the b-trees using this command:

mongo test --eval 'printjson(db.c.find({x:1}).hint({_id:1}).toArray())'

While this is running you should observe the cache utilization climb to its limit, 80%.

Now wait a minute or two (so that the b-trees haven't been accessed in a while), then trigger eviction of the test.c collection and _id index by creating a new test.d collection (20 MB will do):

for t in $(seq 16); do
    mongo test --eval "         
        every = 10000
        for (var i=0; ; ) {
            var bulk = db.d.initializeUnorderedBulkOp();
            for (var j=0; j<every; j++, i++)
                bulk.insert({})
            bulk.execute();
            print(i)
            
            s = db.d.stats(1024*1024)
            if ($t==1)
                print(s.size, s.totalIndexSize, s.size+s.totalIndexSize)
            if (s.size+s.totalIndexSize >= 20)
                break
        }
    "
done

Soon after this (within a minute) the sweep server will begin evicting the test.c collection and _id index b-trees; you will see cache utilization drop from 80% to 0% over the course of some time (a couple minutes, somewhat longer on Windows than Linux). During this time accesses to test.c block.



 Comments   
Comment by Githook User [ 27/Apr/15 ]

Author:

{u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}

Message: Discard trees from cache in the background.

We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles). If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed.

With this change, handles are marked "dead", and readers will no longer use them. The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on.

refs SERVER-17907, SERVER-18014
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46

Comment by Githook User [ 13/Apr/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: Don't sweep unless there's a reason we need to close file handles.
Add a new configuration option, file_manager.open_handles that sets
a minimum number of file handles that must be open before sweep runs,
default is 250. Reference #1856, SERVER-17907.
Branch: cursor-pin
https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

Comment by Githook User [ 10/Apr/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: Don't sweep unless there's a reason we need to close file handles.
Add a new configuration option, file_manager.open_handles that sets
a minimum number of file handles that must be open before sweep runs,
default is 250. Reference #1856, SERVER-17907.
Branch: develop
https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

Comment by Githook User [ 08/Apr/15 ]

Author:

{u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'}

Message: Don't sweep unless there's a reason we need to close file handles.
Add a new configuration option, file_manager.open_handles that sets
a minimum number of file handles that must be open before sweep runs,
default is 250. Reference #1856, SERVER-17907.
Branch: sweep-file-handle-minimum
https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

Generated at Thu Feb 08 03:45:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.