[SERVER-46363] Chunk move failing after removing shard from cluster Created: 24/Feb/20 Updated: 27/Oct/23 Resolved: 04/Mar/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Dilip Kolasani | Assignee: | Carl Champain (Inactive) |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | ALL |
| Participants: |
| Description |
|
Hi we are using mongo sharded cluster running with 4.2.1. Architecture: Since shard1 and shard2 are under utilized, we decided to remove shard2. we did the following steps 1) we issued remove shard from mongos and also Moved databases to another shard out of 2 sharded collections, all chunks related to 1 collections are drained to another shard.
we tried even moving some of chunks manually and they also failed with same reason. sh.status() output is attached We issued the following command to include chunk info from above sh.status() output to move one chunk command:
Output:
Apart from this , we also issued flushRouterConfig multiple times and we restarted all mongos. But still same issue exists. Please let me know if there is any known bug around this or any configuration that we need to tweak on our side. |
| Comments |
| Comment by Carl Champain (Inactive) [ 04/Mar/20 ] | |
|
Thanks for getting back to us! I will now close this ticket. | |
| Comment by Dilip Kolasani [ 03/Mar/20 ] | |
|
Thanks Carl for detailed summary.hypothesis 1 is confirmed. We are working on enforcing the uniqueness of the _id index on application side.we can now close this ticket. | |
| Comment by Carl Champain (Inactive) [ 28/Feb/20 ] | |
|
After investigating your issue, we’ve come up with two hypotheses: 1. The _id index key is not unique across your sharded cluster. Our documentation says the following about the uniqueness of the _id index across a sharded cluster:
So, in your case, we noticed that _id is neither the shard key nor the prefix of the shard key, which makes it possible that a document on shard2 has the same _id as a document on shard1.
To determine whether hypothesis 1 or 2 is correct, please connect directly to the primary replica set member of shard1 and shard2 and run:
Kind regards, | |
| Comment by Dilip Kolasani [ 26/Feb/20 ] | |
|
Please confirm if this is anyway related to https://jira.mongodb.org/browse/SERVER-45844 also? | |
| Comment by Dilip Kolasani [ 25/Feb/20 ] | |
|
Hi Carl, regards | |
| Comment by Carl Champain (Inactive) [ 25/Feb/20 ] | |
|
Thank you for the report.
We've created a secure upload portal for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Kind regards, |