[SERVER-34610] One slow query should not block the whole replica set Created: 23/Apr/18 Updated: 24/Apr/18 Resolved: 24/Apr/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.6.2 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Emilien Kenler [X] | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Last Friday, we encountered an issue around 07:20 UTC (16:20 JST). We would like to have assistance from the MongoDB team to help:
At the time of incident, the primary server was mongodb001-0305.prd.jp1.aws.kkvesper.net, the affected secondary was mongodb001-005a.prd.jp1.aws.kkvesper.net and mongodb001-0c50.prd.jp1.aws.kkvesper.net was an unaffected secondary. We are using MongoDB 3.6.2 with mongoid 3.1.6. The replica set has 3 members connected to the application and one hidden secondary used for backups. We believe the high CPU was caused by a query using the wrong index (see mongodb_parsed_005a_2018042007.log at 2018-04-20T07:18:19.692Z). The CPU usage started to increase soon after the query started, and went down only when we manually restarted the MongoDB server. While the query took 5 minutes, we can see that the high CPU usage lasted more than that. Using mlogvis, it seems a lot of queries were still running when the server has been restarted and that most of the slow queries are those ones. With mloginfo, I can see there are more queries run on the affected secondaries than on the other servers. Is it an issue with mongoid or the server? I would expect all secondaries to receive the same amount of queries. Please provide guidance to analyse this issue and prevent it to happen again. Thank you. |
| Comments |
| Comment by Kelsey Schubert [ 24/Apr/18 ] |
|
Hi MiLk Thanks for your report. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group. Kind regards, |