[SERVER-24194] Queued table drops within the WiredTiger KVEngine can compete with each other Created: 18/May/16  Updated: 22/Nov/16  Resolved: 01/Jun/16

Status: Closed
Project: Core Server
Component/s: Storage, WiredTiger
Affects Version/s: None
Fix Version/s: 3.2.8, 3.3.8

Type: Bug Priority: Major - P3
Reporter: David Hows Assignee: David Hows
Resolution: Done Votes: 0
Labels: code-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to WT-2646 Split the lock_wait flag into two, ad... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Steps To Reproduce:

Run the test jstest/slow1/remove_during_mr.js on OSX or on Linux with strace instrumentation to slow system calls.

Participants:
Linked BF Score: 0

 Description   

Queued table drops within the WTKVEngine can compete with each other.

When creating and dropping a large number of tables (such as running multiple concurrent map/reduce jobs) MongoDB creates a large number of tables then drops them in quick succession. Given that the instance is still "hot" at this point as there are multiple create and drop operations going on at once, many of these drops get queued to be dropped later.

This queuing then triggers a close of all sessions and any number of threads will then enter the code path to drop all of the queued collections all starting at the top. Having multiple threads enter the drop code at once means that we often see several attempts to drop one collection(s) before a single drop succeeds, during which time further collection drops are often queued. This causes an almost quadratic case, where a high rate of collection creates/drops will cause more and more drops to be queued.

One thing needed to create the conditions for this is a small level of delay in processing of the deletes. I suspect this occurs on OSX as we must use the slower fsync or with strace on linux - which slows the OS calls.



 Comments   
Comment by Githook User [ 24/Jun/16 ]

Author:

{u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

Message: SERVER-24194 - Backport, change the dropAllQueued loop to avoid duplicating work over many threads
Branch: v3.2
https://github.com/mongodb/mongo/commit/ba5fe233b5915d535a7b73da911b6b9fe132c035

Comment by Githook User [ 01/Jun/16 ]

Author:

{u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

Message: SERVER-24194 - switch from the WiredTiger lock_wait flag to the checkpoint_wait flag
Branch: master
https://github.com/mongodb/mongo/commit/8b13ccc77395e153a607e49049323d07d7b7f9da

Comment by David Hows [ 26/May/16 ]

I've made an initial commit here of the logical changes to the dropAllQueued function (now renamed dropSomeQueuedIdents). Will need one further change when the changes in WT-2646 are moved upstream into MongoDB.

Comment by Githook User [ 26/May/16 ]

Author:

{u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}

Message: SERVER-24194 - Change the dropAllQueued loop to avoid duplicating work over many threads
Branch: master
https://github.com/mongodb/mongo/commit/cc6ee9cf116853289ee41784220b3ce8ed14c29c

Comment by David Hows [ 18/May/16 ]

The modifications I made to remove locking overlap between new drop's being performed and the dropAllQueued operation running have made the original issue re-appear. Back to the drawing board.

Comment by David Hows [ 18/May/16 ]

Good questions agorrod.

Does this put a greater time burden on a single thread?

Not really. Currently all the threads are walking the whole queue anyway, now it is just one. That one may do slightly more work, as it may now get a chance to acquire the 3 WT locks involved in a drop and actually complete the table drop operation. Another thing of note is that the WT table drop call did not originally have the force and lock_wait=false flags set, these have been set more recently and should mean we never hit a heavyweight operation as part of the drop (that is left to other threads within WT).

Can dropAllQueued be called in an application thread?

Yes. From any thread that uses a WiredTigerSession object

If so, could it end up spending an unfair amount of time doing the drops?

Potentially possible. But I can't envision a scenario which is worse than the current behaviour, as each thread must walk the whole list of tables to drop anyway.

Given that the _identToDropMutex is being held for longer in dropAllQueued - is it a concern that calls to WiredTigerKVEngine::_drop could need to wait for the lock?

I've updated the patch to avoid this problem. Thanks

Are there tests we can run that will give us confidence there won't be negative consequences?

Not sure, but I will try and either find some or write a test case.

Comment by Alexander Gorrod [ 18/May/16 ]

david.hows Thanks - I'm glad you got to the bottom of this failure. I'm concerned that switching to a single threaded approach here will have negative consequences on some workloads. I can think of two potential degradations:

  • Does this put a greater time burden on a single thread? Can dropAllQueued be called in an application thread? If so, could it end up spending an unfair amount of time doing the drops?
  • Given that the _identToDropMutex is being held for longer in dropAllQueued - is it a concern that calls to WiredTigerKVEngine::_drop could need to wait for the lock?

Are there tests we can run that will give us confidence there won't be negative consequences?

Comment by David Hows [ 18/May/16 ]

Two things to consider here in terms of solution:

  1. Single thread the dropAllQueues function to avoid quadratic build up
  2. Look at logic to avoid queuing the cacheing of WT cursors on temp tables.

The cacheing of cursors on temp tables mentioned above is one of the initial conditions that leads to the initial inability to drop a table. If we can remove some of the initial conditions, this will help avoid causing problems.

Generated at Thu Feb 08 04:05:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.