[SERVER-26040] High CPU/IOWAIT in MongoDB 3.2.9 Created: 09/Sep/16 Updated: 12/Dec/16 Resolved: 12/Dec/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.2.9 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Emil Burzo | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
After upgrading a replica set to 3.2.9, I am seeing:
The workload hasn't changed before/after the upgrade (about 40-60 inserts / 1-2 queries / 0 updates / 0 deletes – per second) The configuration also hasn't changed, I've been running WiredTiger before as well. The storage configuration is configured as follows:
I've attached diagnostic.data as requested and some munin charts that show the difference before/after. Seemingly related issues: Let me know if there's anything else I can do to help. |
| Comments |
| Comment by Kelsey Schubert [ 12/Dec/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Hi emilburzo, Your responses indicate that the WT journal flushing is working as designed. As previously discussed, WT journal flushing in MongoDB 3.2 does do more IO than in 3.0. Please open a new ticket if you are experiencing performance issues (e.g. slow queries) as a result of this behavior. Kind regards, | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Emil Burzo [ 28/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
It's a bare metal server from online.net
| ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 25/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Hi emilburzo, Thank you for the update. We have not observed that same impact on hard drives in our testing. To continue to investigate, would you please provide additional details about your environment? Please include a detailed description of your virtualization, filesystem, and storage layer. Thanks again, | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Emil Burzo [ 16/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
I should of tried point 4 before all the others... Disabling the journal did it, so I guess this is a duplicate of I do agree with the submitter though, there should be a more relaxed flush interval setting for less critical workloads. Is there any way to accomplish that currently? Hard drives are +5°C hotter with the current 50ms flush interval. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Emil Burzo [ 15/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Hey Thomas,
Better steps to reproduce: I have tested the following on a (mostly) idle machine running Ubuntu 16.04 with software RAID1 on regular 7k rpm consumer drives. Downloaded the following MongoDB versions:
Unpacked the archive, switched into each version's directory. Made some ad-hoc data dirs:
Started three instances:
Contents of mongo.conf
Initialized the replica set:
Opened top in another terminal and eyeballed the load average and the "x.x wa" field Ran the following query:
Left it running until it gets to 80 000, the load average crept up and the iowait was higher Specific numbers on the (mostly) idle server I tried: 3.9.2:
3.0.12:
Here's a munin graph: http://i.imgur.com/3DhiLyK.png Hope that helps.
Between the 3.2.0 and 3.2.9 versions no, nothing really major – although I haven't left them running for a long time so there is a possibility. But not as major as the difference between 3.0.12 and 3.2.0.
I'm observing this behaviour on both primaries and secondaries. But 2 out of 3 servers are on SSDs, so the issue is not as visible.
Yes and yes. I'm going to try and disable the journal to see if it makes any difference. | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kelsey Schubert [ 13/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
Hi emilburzo, Thank you for opening this ticket and looking into it in more depth over the weekend. We're still investigating this issue, to help would please answer the following questions?
Please be aware that we expect there to be a higher I/O load on secondary nodes in 3.2 as a result of how we sync the journal. To avoid falling behind, secondaries will self-throttle. For additional details about this behavior, please see Thank you, | ||||||||||||||||||||||||||||||||||||||||||||
| Comment by Emil Burzo [ 11/Sep/16 ] | ||||||||||||||||||||||||||||||||||||||||||||
|
I've spent some time trying to further isolate the issue. The 3.2.x releases show varying degrees of iowait%, some bigger, some smaller. The biggest difference is between 3.0.12 (good) and 3.2.0 (bad). Running with 3.0.12 shows an iowait% of zero, as it was before upgrading. I've also tried to run git bisect to pinpoint the exact commit were the regression was introduced, but I couldn't test properly since I am getting stacktraces if I build from commits between r3.0.12 and r3.2.0 – presumably because the code is not in a clean state. If you can create some stable builds, I can try them out in one of the secondaries from the replica set as the issue is reproducible as there's a visible difference in iowait% between them. Or if you have other suggestions I'm happy to help. |