[SERVER-34604] low performance under 3.6.4 with wiredTiger storage Created: 22/Apr/18 Updated: 04/Nov/18 Resolved: 27/Sep/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance |
| Affects Version/s: | 3.6.4 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Adun | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 6 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||
| Description |
|
we use YCSB to stress mongodb 3.6 vs 3.4, storage wiredTiger. We found that 3.6 slower than 3.4 about 50%. We test 5 times, results as below, the number is qps( only insert thest) 3.6.4 | 3.4.14 YCSB's workload: <noformat> readallfields=true readproportion=0 requestdistribution=zipfian mongod's configuration as below: <noformat> cpu = true maxConns = 65535 smallfiles = true we also test: 1. The hardware/network/OS exactly the same(the same machine). |
| Comments |
| Comment by VictorGP [ 28/Sep/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
I would like to try these fixes. Will they be included in 3.6? Lot of people are stuck in 3.4 waiting for these fixes, so it would be nice if they are included in 3.6 | ||||||||||||||||||||||||||||||||||||||||
| Comment by VictorGP [ 28/Sep/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
Are these fixes also related to this issue? https://jira.mongodb.org/browse/SERVER-31694 I understand you have your priorities, but a 10-20% performance degradation in a database is a serious issue. No matter how many good features a database has, if it is not fast, and it seems MongoDB is taking the wrong route. In | ||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 27/Sep/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
We've gone through all the related regressions in our internal performance testing. A number of fixes have gone into the server to address the issues. However, we still observe an 10-20% regression since 3.4. Consequently, we would expect that you will see some improvement if you rerun your experiment with the latest stable release, but there may still see some regression. Currently, we are prioritizing performance work in other areas and do not have additional improvements planned for this issue, so I'm going close this ticket. However, if the regression still reproduces for you and is significant, please let us know and we'll reopen the ticket. Thank you, | ||||||||||||||||||||||||||||||||||||||||
| Comment by Tomaz [ 06/Aug/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
I wonder if the performance regressions I started to see running our unit tests under MongoDB 3.6 are related to this issue. I recently upgraded mongoengine and pymongo in our project (StackStorm/st2) and updated our Python unit tests so they also run under MongoDB 3.6 on Travis (in addition to MongoDB 3.4). Here is an example build - https://travis-ci.org/StackStorm/st2/builds/412602415 (unit tests under 3.4 run 17 minutes vs 24 minutes under 3.6). This issue and timing can be consistently reproduced if I re-run the build. The only thing which has changed is the database version which is used for the unit tests. I didn't have time to really dig in yet, but since nothing else has changed, it makes it seemed that the issue is indeed related to MongoDB 3.6.3. For reference:
Thanks,
| ||||||||||||||||||||||||||||||||||||||||
| Comment by Adun [ 21/Jun/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
sorry for the typo, it is ~10%
| ||||||||||||||||||||||||||||||||||||||||
| Comment by Henrik Ingo (Inactive) [ 20/Jun/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
Hi Adun
Did you mean to say 15%? I compare the averages of the 5 results you've posted in the description, and the difference seems to be 9.5%. Am I missing some other results somewhere? | ||||||||||||||||||||||||||||||||||||||||
| Comment by David Daly [ 02/May/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
Hi Adun, Thank for reporting this issue. Our internal automated performance testing has identified some regressions which may be related to the performance that you're observing in your tests. To most effectively work to improve performance, we're focusing on our investigation against our internal performance benchmarks which are configured to be consistent and stress particular mongod access paths. Once these issues have been resolved, we may be able to revisit this issue to see whether the same fixes resolve this particular performance regression. Kind regards, | ||||||||||||||||||||||||||||||||||||||||
| Comment by Adun [ 27/Apr/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
when the insert ops become very low, the top 4 events in `perf top` are always:
| ||||||||||||||||||||||||||||||||||||||||
| Comment by Adun [ 22/Apr/18 ] | ||||||||||||||||||||||||||||||||||||||||
|
YCSB's workload:
mongod's configuration as below:
1. about 15% performance lost while running with standalone mode without auth |