[SERVER-33960] 3M context switches due to sched_yield Created: 18/Mar/18 Updated: 07/Apr/23 Resolved: 09/May/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.6.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Юрий Соколов | Assignee: | Dmitry Agranat |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
First, excuse me for non informative description During some dumb testing following weird behaviour were observed:
|
| Comments |
| Comment by Dmitry Agranat [ 09/May/18 ] | ||||||||||||||||||||
|
Hi funny_falcon, I have analyzed the tests performed on April 20th. After upgrading to 3.6.4, I do not see any significant difference compared to the test performed on 3.6.3. Same drops in operation throughput and latency, same 70% kernel CPU, high ~2.6M context switches. All this is correlated with checkpoint scrub dirty target and checkpoint blocking page eviction:
Even though there are still periodic drops in operation throughput and latency, after disabling the eviction_checkpoint_target=0, none of the above indicators are present. High kernel CPU was replaced with high iowait and only up to 160K context switches. All most visible changes are related to io, where the bottleneck was moved. There is also some indication that there is a data/journal contention which are both colocated on the same volume:
For improved performance, please review our production notes, specifically, this section. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group. Thanks, | ||||||||||||||||||||
| Comment by Юрий Соколов [ 20/Apr/18 ] | ||||||||||||||||||||
|
Today we encountered issue in production, when we try to shard big collection to two shards. So I'm repeating experiment with Mongo 3.6.4 (to discover: will upgrade help us, or not). Without adding `eviction_checkpoint_target=0` it looks like 3.6.4 starts to spike very early: at 18:32:19 was first spike of context switches. At this moment MongoDB consumed only 7.2GB of memory, while it consumes 13.1GB in peak. Then I stopped mongo (at 18:57), and added `eviction_checkpoint_target=0`, and then started again (19:09). This stalls and spikes are not that huge compared to I saw in previous experiment, but this experiment just started, and I'm not optimistic on its future. I don't see how I can reopen this Issue. Could you do it, please? | ||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 16/Apr/18 ] | ||||||||||||||||||||
|
We haven't heard back from you about the recommended configuration change, so I am going to resolve this ticket. If you have any further issues with excessive yielding, please reopen and follow up. | ||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 22/Mar/18 ] | ||||||||||||||||||||
|
funny_falcon, after examining all of the data that you have attached to this ticket, it looks like the default configuration of checkpoints is causing these inefficient cases. You can disable the code causing this with the following in your config file:
Based on the data you have attached, I would not expect this change to cause problems, but you should verify locally before making this change in production. I am not completely sure about the cause of the segfault based on the information available. I think it could be caused by the bug fixed in | ||||||||||||||||||||
| Comment by Юрий Соколов [ 20/Mar/18 ] | ||||||||||||||||||||
|
I've attached last diagnostic file. | ||||||||||||||||||||
| Comment by Юрий Соколов [ 20/Mar/18 ] | ||||||||||||||||||||
|
today mongod segfaulted. Unfortunately, I didn't tune corepath, so ubuntu's apport (still) tries to write crash to root partition, and I believe there is no enough space for.
| ||||||||||||||||||||
| Comment by Юрий Соколов [ 19/Mar/18 ] | ||||||||||||||||||||
|
I've attached diagnostic_data and stack traces. btw, when issue happening, request latency can be as large as 2 seconds, even for "lookup by _id" . Every "issue" occurrence leads to latency increasing, and some of them are that bad (at least 10 times during today). | ||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 19/Mar/18 ] | ||||||||||||||||||||
|
If this has happened in the last 7 days or so, can you please upload the contents of dbpath/diagnostic.data to this ticket? Also, can you please clarify how many mongod processes are running on this computer? I'm attaching below a small shell script to capture stack traces that you can run if this happens again. This will also provide additional information:
| ||||||||||||||||||||
| Comment by Юрий Соколов [ 18/Mar/18 ] | ||||||||||||||||||||
|
Forgot to mention, that it looks to correlate with io latency on stat screenshot. And, probably, with ttl thread doing its work. |