[SERVER-25060] Make SplitHeuristics.maxParallelSplits tunable Created: 14/Jul/16 Updated: 06/Dec/22 Resolved: 28/Sep/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.2.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | xihui he | Assignee: | [DO NOT USE] Backlog - Sharding Team |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Sharding
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
We continue to observe lock contention during the split in our environment, which is caused by mongos sending 5 split requests on the same chunk at the same time. All the 5 split requests failed due to lock contention, and mongos would try to split again and again. It makes the sharding become very slow. I seems to me it's better to disable to parallel splits? Attached is the detail logs on mongod. |
| Comments |
| Comment by xihui he [ 15/Jul/16 ] |
|
I investigate further and find there are 15 split requests from 3 mongos at the same time, and only one split from one mongos successes. All requests from other two mongos fail, so they keep using stale config and keep trying to split. I have disabled auto split on the other two mongos, and the issue seems mitigated. However the remaining one still sends concurrent split requests (5) to mongod, and it seems to me split request is a heavy operation. It's better to have a workaround to mitigate the issue. Thanks! |
| Comment by xihui he [ 15/Jul/16 ] |
|
renctan Thanks for your reply. I still don't understand why all the splits fail. From the log it shows it finds the split points, but fails in the later steps. Appreciate if you could share your insight. |
| Comment by Randolph Tan [ 14/Jul/16 ] |
|
Hi, Thanks for the report, we are aware of the issue and plan to address this by moving the auto split logic to the shards. You can watch this ticket ( Thanks! |