[SERVER-11509] movePrimary should error when database is not drained Created: 31/Oct/13  Updated: 06/Dec/22  Resolved: 12/Nov/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Walt Woods Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Done Votes: 1
Labels: PM-1017
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu servers


Issue Links:
Depends
depends on SERVER-939 Ability to distribute collections in ... Blocked
Related
related to SERVER-8059 After movePrimary, db.getCollectionNa... Closed
Assigned Teams:
Sharding
Operating System: ALL
Steps To Reproduce:

Unsure. We're spending a little time trying to make a reproduction script, but essentially something along the lines of:

1. Have multiple mongos running (we have 12 or so) and several populated shards (we have 8), with some collections in a database sharded and some collections unsharded.

2. Start draining the shard that is the current primary for the database.

3. While it is draining, run movePrimary to another shard.

4. Query each mongos separately, looking for inconsistent results.

Sprint: Sharding 2018-11-19
Participants:

 Description   

We had very confusing behavior where a collection was reporting one set of documents some of the time, and other results at other times. We deduced (and verified) that this was because non-sharded collections (in a sharded environment) were being accessed on two different shards. What appears to have happened is that at least one of our mongos did not get the movePrimary message, meaning it still believed (and interacted with) data on the old shard.

Now, in our situation, we admissibly committed a faux pas: we ran movePrimary while a shard was draining. I realize that the web docs explicitly state not to do this, but it happened.

It seems that movePrimary either:

A) Doesn't move collections atomically
B) Isn't very forceful about having mongos update their routing tables
C) Doesn't play nicely at all with the balancer when a shard is draining

Or something else I suppose. Either way, it seems reasonable that movePrimary will raise an error (rather than creating inconsistencies) if it truly needs to be ran only when all chunks are moved off of a shard. If it does not actually need that, then there clearly is a bug somewhere that is leading to very confusing and inconsistent errors.



 Comments   
Comment by Esha Maharishi (Inactive) [ 12/Nov/19 ]

Closing this as Gone Away since it was fixed by PM-1051.

Comment by Ori Avtalion [ 24/Dec/18 ]

Still relevant for 3.6+.

Comment by Greg Studer [ 07/Aug/14 ]

> If what you say is necessarily the case, the "movePrimary" command should return an error when trying to movePrimary from a shard that still has chunks remaining, rather than silently cause inconsistency.

Reading the history here again, I'm not sure this is the problem you ran into, as you mention above:

> non-sharded collections (in a sharded environment) were being accessed on two different shards

MovePrimary cannot atomically move unsharded collections - the only way to atomically move collections between shards is to shard the collection and move chunk-by-chunk. MovePrimary should not move chunks from sharded collections, and from your description it seems this is not what happened. After a movePrimary, because unsharded collections are not versioned, there is no automated way for mongos to learn of the move aside from a manual flushRouterConfig.

This is definitely an area we're trying to improve, but requires some serious work - see SERVER-939.

Comment by Walt Woods [ 07/Nov/13 ]

Thank you

Comment by David Storch [ 07/Nov/13 ]

Hi Walt, yes, perhaps we should change the error behavior of move primary so that it prevents you from using it when it could be unsafe. I'll update the title to reflect that this is about specifically about error behavior and re-open for triage.

Comment by Walt Woods [ 07/Nov/13 ]

Hi David, that's not really sufficient, as I mention in the bug report.

If what you say is necessarily the case, the "movePrimary" command should return an error when trying to movePrimary from a shard that still has chunks remaining, rather than silently cause inconsistency.

Comment by David Storch [ 07/Nov/13 ]

Hi Walt,

As you have noticed in the documentation, the movePrimary command should only be used 1) after all sharded collections have been drained from the database, or 2) the database does not contain any collections with data. I'm going to mark this ticket "Works as Designed", as the movePrimary command is only designed for use in these limited circumstances.

Using movePrimary in other circumstances is neither atomic nor isolated and can lead to data inconsistency problems such as the one that you experienced. Moving unsharded collections (and, consequently, moving databases which contain unsharded collections) is difficult to do safely because MongoDB does not track metadata for such collections.

Comment by Walt Woods [ 05/Nov/13 ]

We didn't have too much time, but were unable to reproduce this via a simple script. Definitely still a concerning issue though.

Generated at Thu Feb 08 03:25:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.