[SERVER-32418] Due to a lag, query is very slow in the config server Created: 20/Dec/17 Updated: 27/Oct/23 Resolved: 05/Mar/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, Sharding |
| Affects Version/s: | 3.4.10 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | liangzhang | Assignee: | Dmitry Agranat |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
System environment The config server has three members in my cluster, one of members that is secondary shows some slow queries in logs file. When I executed sh.disableAutoSplit(), there was more slow queries on this secondary's log. And the ops of all members have risen to 2k/s.
|
| Comments |
| Comment by Dmitry Agranat [ 04/Mar/18 ] | ||
|
Thank you for providing the requested information. As suspected, when there is a lag, the reported query is slow in the config server. This is expected as explained in my previous comment. Looking at the two such occurrences:
I am going to close this ticket as "works as designed". However, I'd like to mention one abnormality I've noticed:
The practice of mixing different major MongoDB versions in a replica set is not recommended (except for the time of the upgrade). Perhaps, by fixing this misconfiguration, you might fix the issue with the periodic lag in your config servers replica set. Thanks, | ||
| Comment by liangzhang [ 02/Mar/18 ] | ||
|
files have been uploaded . | ||
| Comment by Dmitry Agranat [ 28/Feb/18 ] | ||
|
Thank you for providing the requested information. I'd like to validate that there is no replication lag between your config servers. Please note that the read is doing afterOptime in combination with readConcern: majority so the query will block, holding no locks, until a majority of members have reached the specified optime.
Thanks, | ||
| Comment by liangzhang [ 23/Feb/18 ] | ||
|
Sorry, the reply is delayed. Uploaded log of all nodes.All nodes have this problem. | ||
| Comment by Kelsey Schubert [ 16/Feb/18 ] | ||
|
We still need additional information to diagnose the problem. If this is still an issue for you, would you please address the questions in Mark's most recent comment? Thank you, | ||
| Comment by Mark Agarunov [ 23/Jan/18 ] | ||
|
Hello ufozhangliang@163.com, Thank you for the additional information and my apologies for the delay in replying. Unfortunately we have not yet been able to determine the cause of this behavior. If possible, could you provide the complete logs from all nodes in the cluster? Additionally, is this behavior causing adverse issues such as performance regressions? Do all the nodes experience the same issue or only this specific node? Thanks, | ||
| Comment by liangzhang [ 27/Dec/17 ] | ||
|
if re-enable autosplit, this slow query will also have less than disabled, but there will also be. | ||
| Comment by Mark Agarunov [ 26/Dec/17 ] | ||
|
Hello ufozhangliang@163.com, Thank you for providing this information. After looking over the logs and diagnostic data, it seems that the slow queries are correlated with saslStart and saslContinue commands: This leads me to suspect that the slow queries may be due to a slow response from the sasl authentication. Was this behavior present in any capacity before disabling autosplitting, and does it continue if autosplit is re enabled? Thanks | ||
| Comment by Mark Agarunov [ 21/Dec/17 ] | ||
|
Hello ufozhangliang@163.com, Thank you for the report. To get a better idea of what may be causing this behavior, could you please provide the following:
This should give some insight into this issue. Thanks, | ||
| Comment by liangzhang [ 20/Dec/17 ] | ||
|
config server queries take about 2s, and sometimes it would block our client query. |