[SERVER-26674] Over 25% regression on mongodb using YCSB workload Created: 18/Oct/16 Updated: 19/Nov/16 Resolved: 08/Nov/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance |
| Affects Version/s: | 3.3.10, 3.3.12, 3.3.15, 3.4.0-rc0 |
| Fix Version/s: | 3.4.0-rc3 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Lilian Romero | Assignee: | Samantha Ritter (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | use YCSB 0.6.0 workload with 12 threads |
||||||||
| Sprint: | Platforms 2016-10-31, Platforms 2016-11-21 | ||||||||
| Participants: | |||||||||
| Description |
|
Over 25% regression on mongodb after release 3.3.9.
Here are the results.
|
| Comments |
| Comment by Samantha Ritter (Inactive) [ 08/Nov/16 ] | ||||||||||||||||||||||||||||
|
Thank you for reporting this performance regression. I've done some refactoring in our TransportLayer code so we can avoid time spent under locks. Also, I have reduced the number of lookups we need to do to access our connection objects. Running a similar YCSB workload to yours by hand on my machine, I am seeing significantly higher throughput with this fix. If you get the chance to run your particular workload on the same server that you were using before, I'd be very interested to see what results you get. For now, I am going to close this ticket, and the patch will be going into 3.4.0-rc3. If you observe other regressions or if your workload does not perform better with this fix, please do open another ticket. Thanks, | ||||||||||||||||||||||||||||
| Comment by Githook User [ 08/Nov/16 ] | ||||||||||||||||||||||||||||
|
Author: {u'username': u'samantharitter', u'name': u'samantharitter', u'email': u'samantha.ritter@10gen.com'}Message:
| ||||||||||||||||||||||||||||
| Comment by Githook User [ 08/Nov/16 ] | ||||||||||||||||||||||||||||
|
Author: {u'username': u'samantharitter', u'name': u'samantharitter', u'email': u'samantha.ritter@10gen.com'}Message: | ||||||||||||||||||||||||||||
| Comment by Githook User [ 06/Nov/16 ] | ||||||||||||||||||||||||||||
|
Author: {u'username': u'samantharitter', u'name': u'samantharitter', u'email': u'samantha.ritter@10gen.com'}Message: | ||||||||||||||||||||||||||||
| Comment by Githook User [ 06/Nov/16 ] | ||||||||||||||||||||||||||||
|
Author: {u'username': u'samantharitter', u'name': u'samantharitter', u'email': u'samantha.ritter@10gen.com'}Message: | ||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 25/Oct/16 ] | ||||||||||||||||||||||||||||
|
We're not seeing this on other platforms, but looks likely to be a result of transport layer changes during that time frame. Thanks for the report. | ||||||||||||||||||||||||||||
| Comment by Jim Van Fleet [ 24/Oct/16 ] | ||||||||||||||||||||||||||||
|
from pert_tree.prof (perf record -g ; perf report -vg ) – a pretty clear trail from _raw_spin_lock back to mongo::transport::TransportLayerLegacy::_runTicket I've seen the sharding and cursor manager locks be hot in the past – I think this new one is overshadowing them – note in the data below that the cursorManager and sharding are calling pthread_mutex_lock, but the percentages are pretty low –
| ||||||||||||||||||||||||||||
| Comment by Chung-yen Chang [ 19/Oct/16 ] | ||||||||||||||||||||||||||||
|
lilianr@us.ibm.com, thanks for the information. We are looking into this and will get back to you with our findings. | ||||||||||||||||||||||||||||
| Comment by Lilian Romero [ 19/Oct/16 ] | ||||||||||||||||||||||||||||
|
here are the options used: | ||||||||||||||||||||||||||||
| Comment by Chung-yen Chang [ 19/Oct/16 ] | ||||||||||||||||||||||||||||
|
lilianr@us.ibm.com, thanks for sharing the data with us. Would you please tell me what is the read/update mix of the YCSB workload that you used? YCSB has multiple sub-workloads with different read/update mix and I am trying to see if this could be related to a particular op type. Thanks. | ||||||||||||||||||||||||||||
| Comment by Lilian Romero [ 18/Oct/16 ] | ||||||||||||||||||||||||||||
|
The workload used is YCSB 0.6.0. Number of threads used: 12. | ||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 18/Oct/16 ] | ||||||||||||||||||||||||||||
|
Thanks for the report Lilian, we'll take a look. Can you please provide
? Thanks, | ||||||||||||||||||||||||||||
| Comment by Lilian Romero [ 18/Oct/16 ] | ||||||||||||||||||||||||||||
|
stack trace:
|