[SERVER-38074] abnormal performance downgrade and large traffice between primary shard and config server Created: 10/Nov/18 Updated: 12/Nov/18 Resolved: 12/Nov/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.2.20 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Adun | Assignee: | Danny Hatcher (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
Mongo Version 3.2.20, clusters deployed as: 15 or more shard replica sets + 1 config replica sets. Each shard has three nodes(1 primary + 2 secondary), the storage is mmapv1. Abnormal performance downgrade occur irregularly: 1. The traffics output from all config nodes are very large. log a:
log b:
4. at this time, large slow logs were generated on primary node, the op time were growing, db.currentOp() on primary shard, shows that many queries were `"msg" : "waiting for write concern",` 5. after we rise the loglevel, we found large FIND QUERY for chunks collection from primary node to config servers.
6. This situation can be solved temporarily through stepDown config primary or stopDown primary shard. But it will happen again in the uncertain time of the future. |
| Comments |
| Comment by Adun [ 12/Nov/18 ] |
|
config servers use WiredTiger as storage engine. shard servers use MMAPv1 as storeage engine. It occured irregularly. The cluster are available most of the time. |
| Comment by Danny Hatcher (Inactive) [ 12/Nov/18 ] |
|
Hello Adun, You mention in your initial description that you have config servers deployed in a replica set but you are using MMAPv1 as a storage engine. Please note that when deployed in a replica set, config servers must be using the WiredTiger storage engine. Your current issue may be the result of config server unavailability. Config servers must be available at all times in order to ensure the normal operation of a sharded cluster. For further MongoDB-related support discussion, please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group. Thank you, Danny |