[SERVER-20996] 74% performance regression in db.c.remove() Created: 18/Oct/15 Updated: 30/Mar/16 Resolved: 19/Nov/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, WiredTiger |
| Affects Version/s: | 3.2.0-rc0 |
| Fix Version/s: | 3.2.0-rc4 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Bruce Lucas (Inactive) | Assignee: | David Hows |
| Resolution: | Done | Votes: | 0 |
| Labels: | WTplaybook | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
74% performance regression in db.c.remove() Test inserts n empty documents into a collection, then removes them with db.c.remove({}). Timing:
The rate of document removals under 3.2.0-rc0 is initially lower and also declines over time (shown below), whereas under 3.0.7 it started higher and remained constant.
I observed on a longer run that the rate of removal continued to decline, so would expect the size of the regression to increase with a larger number of documents, continuing the trend seen with the two collection sizes measured above. Stack traces show significant time spent walking the tree, so perhaps this is related to this issue. Repro script:
|
| Comments |
| Comment by Githook User [ 19/Nov/15 ] | ||||||||||||
|
Author: {u'username': u'daveh86', u'name': u'David Hows', u'email': u'howsdav@gmail.com'}Message: | ||||||||||||
| Comment by David Hows [ 17/Nov/15 ] | ||||||||||||
|
Just finished the bisect of the second part of the regressions from 50 sec to 100 sec. Looks like the following drop from WT into Mongo was the cause. This drop also includes a pull from wt develop into the wt mongodb-3.2 branch, so lots of threads to pull apart.
| ||||||||||||
| Comment by Alexander Gorrod [ 17/Nov/15 ] | ||||||||||||
|
david.hows could you re-run your test removing a varied proportion of the content? i.e: 5%, 15%, 25%, ...95%. It would be interesting to know whether the volume of removes is relevant. | ||||||||||||
| Comment by David Hows [ 17/Nov/15 ] | ||||||||||||
|
As per a suggestion from Michael, I have made a second workload that looks to remove 5% of documents from the collection randomly. In 3.0.7 we get
In 3.2.0 we get
So, this looks like randomised removes themselves aren't a problem and its only the full remove of all documents that is in issue here. | ||||||||||||
| Comment by David Hows [ 17/Nov/15 ] | ||||||||||||
|
According to a git bisection, the first half of this regression (between 3.1.4 and 3.1.5) was introduced by
I will chase down the other half now. | ||||||||||||
| Comment by David Hows [ 16/Nov/15 ] | ||||||||||||
|
You were right on
__wt_split_reverse was even worse again
I wasn't able to trivially make the changes to __wt_page_evict_soon that didn't freeze MongoD. I've also run against 3.1.5 just to see what the impact was only a 5x regression. Seems like there may be more than one thing at play here.
Final testing shows 3.1.4 looks like this, so I suspect that the regression may have started in 3.1.4
| ||||||||||||
| Comment by Michael Cahill (Inactive) [ 16/Nov/15 ] | ||||||||||||
|
daveh86, can you please try a few things? The branch for A big change was Similarly, try turning off the call to __wt_split_reverse in __evict_delete_ref to see whether that is involved. Thanks! | ||||||||||||
| Comment by David Hows [ 16/Nov/15 ] | ||||||||||||
|
Looks like it relates to the MongoD binary, rather than the shell.
3.2 shell w/ 3.0.7 mongod
| ||||||||||||
| Comment by Daniel Pasette (Inactive) [ 16/Nov/15 ] | ||||||||||||
|
Can you double check it's not a shell diff? Unlikely, but should verify. | ||||||||||||
| Comment by David Hows [ 16/Nov/15 ] | ||||||||||||
|
michael.cahill still looks like there is a pretty big regression. 80~ seconds for RC2 vs 10~ seconds for 3.0.7
| ||||||||||||
| Comment by Michael Cahill (Inactive) [ 12/Nov/15 ] | ||||||||||||
|
daveh86, can you please retest with the tip of master, now that | ||||||||||||
| Comment by Daniel Pasette (Inactive) [ 19/Oct/15 ] | ||||||||||||
|
Ran the repro on a series of 3.1 dev series + 3.0.7: MongoDB shell version: 3.2.0-rc1-pre- MongoDB shell version: 3.2.0-rc1-pre- MongoDB shell version: 3.2.0-rc1-pre- MongoDB shell version: 3.2.0-rc1-pre- MongoDB shell version: 3.2.0-rc1-pre- |