[SERVER-18512] Performance drop after migrating from 2.4 to 2.6 Created: 18/May/15 Updated: 28/Oct/15 Resolved: 28/Oct/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.6.8, 2.6.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Marzio Rizzo | Assignee: | Sam Kleinman (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Immediately after migrating a mongodb replica set from 2.4.5 to 2.6.9 we observed an increased lock % on all members, even on secondaries that were only synchronizing without serving queries. |
| Comments |
| Comment by Ramon Fernandez Marina [ 28/Oct/15 ] |
|
Hi mrizzo, apologies for the radio silence. We have been unable to reproduce this behavior on our end after trying in different environments, and we're not sure what may be different about your environment that causes the performance drop you report, so unfortunately there's not much more investigation we can do on our end. That being said, MongoDB 3.0 includes numerous improvements over 2.6, including collection-level locking for MMAPv1, so I'd encourage you to try out 3.0.7 (the latest stable release at the time of this writing) and see if it performs better than 2.4/2.6 for your use case. Regards, |
| Comment by Marzio Rizzo [ 15/Jun/15 ] |
|
Hi ramon.fernandez, 1. Install mongo-2.4.5 on all 3 nodes We attached the mongo_replicaset_test_results.xlsx report with our results. |
| Comment by Marzio Rizzo [ 05/Jun/15 ] |
|
Hi ramon.fernandez, could you please provide us the mongodb configuration that you're using during your tests? Thanks, |
| Comment by Ramon Fernandez Marina [ 29/May/15 ] |
|
mrizzo, I have not been able to reproduce this behavior on my end, and in fact I'm observing the opposite effect: in 2.4.5 the lock% hovers around 70% and I get around 75K updates/second, while in 2.6.9 the lock% hovers around 50% and I get around 155K updates/second. I looked at the logs you sent and they just contain startup data, so not much information there that can help us understand what your replica set is doing. I also looked at your MMS graphs and I don't see anything strange happening from 9-11am (except for the lock% difference). While I do see the data from mongostat that you obtained I haven't been able to observe the same behavior. Is there other information you may be able to provide that can be of help here? Are you using any other options for your {{mongod}}s like powerOf2Sizes, noprealloc or smallfiles by any chance? Anything else you can think of? Thanks, |
| Comment by Ramon Fernandez Marina [ 28/May/15 ] |
|
mrizzo, we haven't been able to reproduce this behavior as of yet but we continue to investigate. We'll post updates to this ticket when we have more information. Cheers, |
| Comment by Marzio Rizzo [ 28/May/15 ] |
|
Hi guys, do you need any further information/material? Did you find useful the attached inputs? Thanks, best regards |
| Comment by Marzio Rizzo [ 22/May/15 ] |
|
We forgot to attach mongostat output in which you can see the lock % of all mongod during the load test (ended at 16:53:07). |
| Comment by Marzio Rizzo [ 22/May/15 ] |
|
We attached the requested logs. Let us know if you need more details. |
| Comment by Ramon Fernandez Marina [ 21/May/15 ] |
|
Can you please upload the log files for the primary and the 2.6.9 secondary from startup until you run your load test? The "from startup" part is to see how are your mongod configured, but if the logs are too large you can upload the part corresponding to the load test and the configuration files used for each mongod. Thanks, |
| Comment by Marzio Rizzo [ 19/May/15 ] |
|
OS is Centos 6.5 64 bit with kernel 2.6.32-431.el6.x86_64.
|