[SERVER-19445] Performance regression in Mongo-Perf mms tests with wiredTiger Created: 16/Jul/15 Updated: 05/Feb/16 Resolved: 20/Jul/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, WiredTiger |
| Affects Version/s: | 3.1.5 |
| Fix Version/s: | 3.1.6 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | David Daly | Assignee: | Michael Cahill (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | mpreg | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Steps To Reproduce: | In the shell
The update line in the output shows the update throughput. |
||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Description |
|
Running mongo-perf test Update.MmsIncShallow1 and the other mms tests shows a large regression (~90%) from master to 3.0.4. On evergreen, 3.1.5 gets 1100 ops/s and 3.0.4 gets 11,434. The following data captures both runs. The first run (A to B) is with 3.0.4. The second run (C to D) is with 3.1.5. |
| Comments |
| Comment by Michael Cahill (Inactive) [ 17/Jul/15 ] | ||||||||
|
For posterity, what was going on in this workload was single-threaded updates to a single document. The document was moderately large, so if WiredTiger accumulated enough versions of the document, it would hit the 10MB limit on the size of a page in memory and forcibly evict the page. There was a change in the drop for 3.1.5 that made WiredTiger lazier about updating the oldest transaction ID in the system. This value determines how many version of the document we think need to stay in cache. We used to update that value actively every time a transaction started, but some workloads go faster if regular operations don't update that shared state. The unfortunate consequence of that change was to trigger eviction in this workload – the oldest ID was lagging far enough behind that we kept around enough versions of the document to reach the 10MB limit. The solution committed into WiredTiger develop is to bump that oldest transaction ID when a transaction commits, if it thought it was the oldest one running. This isn't exact, but particularly for single-threaded workloads it should have the effect of having the oldest ID track the current ID much more closely, which means we throw old versions of the document away much more quickly and never cause unnecessary eviction. | ||||||||
| Comment by Githook User [ 17/Jul/15 ] | ||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: This will help keep the oldest tracked transation ID from falling too far | ||||||||
| Comment by David Daly [ 17/Jul/15 ] | ||||||||
|
I should add that this is a single threaded test workload, updating one document, so I wouldn't expect a need for a lot of sessions or concurrency. | ||||||||
| Comment by David Daly [ 17/Jul/15 ] | ||||||||
|
chung-yen.chang It's a pretty dramatic change on that commit (order of magnitude). On my box it went from ~10.5k ops/sec to ~900 ops/s. Every commit I tested was either > 10k or < 1k. | ||||||||
| Comment by Alexander Gorrod [ 17/Jul/15 ] | ||||||||
|
michael.cahill I'll assign this one to you for now. The commit was a merge with WiredTiger which contains a lot of changes. The graph is interesting - one of the oddest things is that the open session count has dropped from 70 down to near 0. Maybe we are succeeding in forced eviction now, when we were doing in-memory splits before. Assign it back to me if I should investigate further. | ||||||||
| Comment by Chung-yen Chang [ 17/Jul/15 ] | ||||||||
|
david.daly, how much of the original drop can be attributed to git c8310411? Is that pretty much it or there are other contributors that need to be sorted through? | ||||||||
| Comment by David Daly [ 17/Jul/15 ] | ||||||||
|
agorrod Assigning to you for now since you are on the commit. Please let me know if that is okay. | ||||||||
| Comment by David Daly [ 17/Jul/15 ] | ||||||||
|
Bisect done:
| ||||||||
| Comment by David Daly [ 16/Jul/15 ] | ||||||||
|
I will go ahead and do a bisect on this to see if it can be narrowed down to a commit. |