[SERVER-24194] Queued table drops within the WiredTiger KVEngine can compete with each other Created: 18/May/16 Updated: 22/Nov/16 Resolved: 01/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.8, 3.3.8 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | David Hows | Assignee: | David Hows |
| Resolution: | Done | Votes: | 0 |
| Labels: | code-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Completed: | |||||||||
| Steps To Reproduce: | Run the test jstest/slow1/remove_during_mr.js on OSX or on Linux with strace instrumentation to slow system calls. |
||||||||
| Participants: | |||||||||
| Linked BF Score: | 0 | ||||||||
| Description |
|
Queued table drops within the WTKVEngine can compete with each other. When creating and dropping a large number of tables (such as running multiple concurrent map/reduce jobs) MongoDB creates a large number of tables then drops them in quick succession. Given that the instance is still "hot" at this point as there are multiple create and drop operations going on at once, many of these drops get queued to be dropped later. This queuing then triggers a close of all sessions and any number of threads will then enter the code path to drop all of the queued collections all starting at the top. Having multiple threads enter the drop code at once means that we often see several attempts to drop one collection(s) before a single drop succeeds, during which time further collection drops are often queued. This causes an almost quadratic case, where a high rate of collection creates/drops will cause more and more drops to be queued. One thing needed to create the conditions for this is a small level of delay in processing of the deletes. I suspect this occurs on OSX as we must use the slower fsync or with strace on linux - which slows the OS calls. |
| Comments |
| Comment by Githook User [ 24/Jun/16 ] |
|
Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}Message: |
| Comment by Githook User [ 01/Jun/16 ] |
|
Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}Message: |
| Comment by David Hows [ 26/May/16 ] |
|
I've made an initial commit here of the logical changes to the dropAllQueued function (now renamed dropSomeQueuedIdents). Will need one further change when the changes in |
| Comment by Githook User [ 26/May/16 ] |
|
Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}Message: |
| Comment by David Hows [ 18/May/16 ] |
|
The modifications I made to remove locking overlap between new drop's being performed and the dropAllQueued operation running have made the original issue re-appear. Back to the drawing board. |
| Comment by David Hows [ 18/May/16 ] |
|
Good questions agorrod.
Not really. Currently all the threads are walking the whole queue anyway, now it is just one. That one may do slightly more work, as it may now get a chance to acquire the 3 WT locks involved in a drop and actually complete the table drop operation. Another thing of note is that the WT table drop call did not originally have the force and lock_wait=false flags set, these have been set more recently and should mean we never hit a heavyweight operation as part of the drop (that is left to other threads within WT).
Yes. From any thread that uses a WiredTigerSession object
Potentially possible. But I can't envision a scenario which is worse than the current behaviour, as each thread must walk the whole list of tables to drop anyway.
I've updated the patch to avoid this problem. Thanks
Not sure, but I will try and either find some or write a test case. |
| Comment by Alexander Gorrod [ 18/May/16 ] |
|
david.hows Thanks - I'm glad you got to the bottom of this failure. I'm concerned that switching to a single threaded approach here will have negative consequences on some workloads. I can think of two potential degradations:
Are there tests we can run that will give us confidence there won't be negative consequences? |
| Comment by David Hows [ 18/May/16 ] |
|
Two things to consider here in terms of solution:
The cacheing of cursors on temp tables mentioned above is one of the initial conditions that leads to the initial inability to drop a table. If we can remove some of the initial conditions, this will help avoid causing problems. |