[SERVER-11509] movePrimary should error when database is not drained Created: 31/Oct/13 Updated: 06/Dec/22 Resolved: 12/Nov/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.4.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Walt Woods | Assignee: | [DO NOT USE] Backlog - Sharding Team |
| Resolution: | Done | Votes: | 1 |
| Labels: | PM-1017 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu servers |
||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Sharding
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Steps To Reproduce: | Unsure. We're spending a little time trying to make a reproduction script, but essentially something along the lines of: 1. Have multiple mongos running (we have 12 or so) and several populated shards (we have 8), with some collections in a database sharded and some collections unsharded. 2. Start draining the shard that is the current primary for the database. 3. While it is draining, run movePrimary to another shard. 4. Query each mongos separately, looking for inconsistent results. |
||||||||||||||||
| Sprint: | Sharding 2018-11-19 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
We had very confusing behavior where a collection was reporting one set of documents some of the time, and other results at other times. We deduced (and verified) that this was because non-sharded collections (in a sharded environment) were being accessed on two different shards. What appears to have happened is that at least one of our mongos did not get the movePrimary message, meaning it still believed (and interacted with) data on the old shard. Now, in our situation, we admissibly committed a faux pas: we ran movePrimary while a shard was draining. I realize that the web docs explicitly state not to do this, but it happened. It seems that movePrimary either: A) Doesn't move collections atomically Or something else I suppose. Either way, it seems reasonable that movePrimary will raise an error (rather than creating inconsistencies) if it truly needs to be ran only when all chunks are moved off of a shard. If it does not actually need that, then there clearly is a bug somewhere that is leading to very confusing and inconsistent errors. |
| Comments |
| Comment by Esha Maharishi (Inactive) [ 12/Nov/19 ] |
|
Closing this as Gone Away since it was fixed by PM-1051. |
| Comment by Ori Avtalion [ 24/Dec/18 ] |
|
Still relevant for 3.6+. |
| Comment by Greg Studer [ 07/Aug/14 ] |
|
> If what you say is necessarily the case, the "movePrimary" command should return an error when trying to movePrimary from a shard that still has chunks remaining, rather than silently cause inconsistency. Reading the history here again, I'm not sure this is the problem you ran into, as you mention above: > non-sharded collections (in a sharded environment) were being accessed on two different shards MovePrimary cannot atomically move unsharded collections - the only way to atomically move collections between shards is to shard the collection and move chunk-by-chunk. MovePrimary should not move chunks from sharded collections, and from your description it seems this is not what happened. After a movePrimary, because unsharded collections are not versioned, there is no automated way for mongos to learn of the move aside from a manual flushRouterConfig. This is definitely an area we're trying to improve, but requires some serious work - see SERVER-939. |
| Comment by Walt Woods [ 07/Nov/13 ] |
|
Thank you |
| Comment by David Storch [ 07/Nov/13 ] |
|
Hi Walt, yes, perhaps we should change the error behavior of move primary so that it prevents you from using it when it could be unsafe. I'll update the title to reflect that this is about specifically about error behavior and re-open for triage. |
| Comment by Walt Woods [ 07/Nov/13 ] |
|
Hi David, that's not really sufficient, as I mention in the bug report. If what you say is necessarily the case, the "movePrimary" command should return an error when trying to movePrimary from a shard that still has chunks remaining, rather than silently cause inconsistency. |
| Comment by David Storch [ 07/Nov/13 ] |
|
Hi Walt, As you have noticed in the documentation, the movePrimary command should only be used 1) after all sharded collections have been drained from the database, or 2) the database does not contain any collections with data. I'm going to mark this ticket "Works as Designed", as the movePrimary command is only designed for use in these limited circumstances. Using movePrimary in other circumstances is neither atomic nor isolated and can lead to data inconsistency problems such as the one that you experienced. Moving unsharded collections (and, consequently, moving databases which contain unsharded collections) is difficult to do safely because MongoDB does not track metadata for such collections. |
| Comment by Walt Woods [ 05/Nov/13 ] |
|
We didn't have too much time, but were unable to reproduce this via a simple script. Definitely still a concerning issue though. |