[SERVER-30517] Mongos are failing to Calculate Config Difference Created: 04/Aug/17 Updated: 20/Sep/17 Resolved: 11/Aug/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.0.7 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Mohamed Abada | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Participants: |
| Description |
|
Greetings, I hope you're doing well. I am currently facing the below error and I will appreciate it if someone can help or advise how can I get it fixed :
What I am was trying to do is to downscale the cluster and remove one shard, draining completed successfully and all chunks got moved to the primary shard, however after I issued the second shard remove command
I tried db.adminCommand("flushRouterConfig") but it didn't help. Any idea what caused this issue and how can I get it resolved. ? Best Regards, |
| Comments |
| Comment by Mohamed Abada [ 18/Aug/17 ] | ||||
|
Thanks a lot Thomas Schubert for your support on this issue. I would like to confirm that adding shard manually from config server side WORKED successfully. Below you can find what I did exactly in-case some-one else faced the same issue : 1- At the 3 config server I add the shard as following : ```use config; );``` Note maxSize : 1 was used for blancer not to distribute any chunks to Shard2. 2- At Mongos side : ); , "shard1")``` 4- Drained Shard2. )``` I will consider the upgrade ofcourse to avoid such issues in the future. Thanks a lot again. Best Regards, | ||||
| Comment by Kelsey Schubert [ 11/Aug/17 ] | ||||
|
Hi muhamed.abada, Thank you for providing the logs. To resolve this issue, I would recommend manually adding the shard and removing it again:
As I mentioned earlier, we have put significant effort into improving sharding behavior. Therefore, I would recommend upgrading to MongoDB 3.2 and upgrading to CSRS to take advantage of these improvements. Kind regards, | ||||
| Comment by Mohamed Abada [ 11/Aug/17 ] | ||||
|
Dear Thomas Schubert, Any update from your side about the logs ? Also is it possible to drop this collection and restore it from backup ? Best Regards, | ||||
| Comment by Mohamed Abada [ 08/Aug/17 ] | ||||
|
Hi ~Thomas Schubert, Thanks a lot for your feedback and for the time you're allocating to this issue. Attached you can find Shard2 Primary logs requested. As a note because the log file is quit big , Draining finished at 21 of July. Let me know if any additional information is required. Best Regards, | ||||
| Comment by Kelsey Schubert [ 07/Aug/17 ] | ||||
|
Hi muhamed.abada, Thanks for clarifying the affected version. In MongoDB 3.2 and later we have introduced stricter checks which may prevent this type of situation from reoccurring (e.g. Since restarting the mongos did not resolve the issue, would you please provide the complete logs from the primary of shard2 so we can better understand what happened here? Thank you, | ||||
| Comment by Mohamed Abada [ 07/Aug/17 ] | ||||
|
Hi Thomas Schubert, Any update here ? Best Regards, | ||||
| Comment by Mohamed Abada [ 04/Aug/17 ] | ||||
|
Hi Thomas Schubert, One note from my side as I tried the below : 1- Backed up everything. And issue persist, not sure how it this chunk got moved to shard2 and how it got removed or drained awhile it contain 1 chunk. Best Regards, | ||||
| Comment by Mohamed Abada [ 04/Aug/17 ] | ||||
|
Hi @Thomas Schubert, version v3.0.7 Best Regards, | ||||
| Comment by Kelsey Schubert [ 04/Aug/17 ] | ||||
|
Hi muhamed.abada, Would you please clarify which version of MongoDB you are using? Thank you, |