[SERVER-4804] db.dropDatabase() through mongos while a migration is happening can lead to wrong "show dbs" output Created: 28/Jan/12 Updated: 06/Apr/23 Resolved: 27/Jun/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Daniel Crosta | Assignee: | Kaloian Manassiev |
| Resolution: | Done | Votes: | 9 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
repro'd on OS X 64bit and CentOS 64bit |
||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Sprint: | Sharding 2019-07-01 | ||||||||
| Participants: | |||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
Steps to reproduce: 1. Bring up a sharded cluster with at least 2 shards
4. Interrupt the data load script, or wait until it completes.
5.b. I immediately retried and got:
5.c. The third try worked:
6. Run "show dbs" again:
7. Verify that data files are missing for all shards (in my case they were) )"
Expected output:
|
| Comments |
| Comment by Kaloian Manassiev [ 27/Jun/19 ] | ||||||||||||||||
|
Starting with version 3.4, the dropDatabase operation properly drops all collections one-by-one and acquires the collection distributed lock, so it will serialize with chunk migration. This is is different from the 3.2 implementation, which directly dropped the database entries from the config server metadata. Please note that due to | ||||||||||||||||
| Comment by Stefan Stark [ 26/Oct/16 ] | ||||||||||||||||
|
Interessting, this might be an authorization problem then. Because I can use the db.dropDatabase() command with the built-in clusterAdmin role, but I cannot listCollections or dropIndex. (https://docs.mongodb.com/manual/reference/built-in-roles/#cluster-administration-roles) | ||||||||||||||||
| Comment by Dave Muysson [ 26/Oct/16 ] | ||||||||||||||||
|
We've worked out a process to give us the greatest chance of success for dropping databases. Since implementing it, we haven't had a failed/ghost database yet. One of our guys dug through the code and found that at some point the DB drop process issued a call to drop an index, but didn't wait/check for success. He thinks that a failure of this action could lead to a partial (and failed) drop. Once we started dropping the indexes ourselves (which sometimes fails, and then works upon retry), our DB's are dropped without issue. Hope this helps others out there. | ||||||||||||||||
| Comment by Stefan Stark [ 26/Oct/16 ] | ||||||||||||||||
|
still exists in 3.2.10 on windows server 2012 | ||||||||||||||||
| Comment by Dave Muysson [ 18/Sep/15 ] | ||||||||||||||||
|
Same problem, Ubuntu 12.04 LTS, mongodb 2.4.14 and 2.6.10 (two different environments). Following the workaround mentioned above at stackoverflow.com did not work for us either. The 2.4.14 cluster was fully restarted following the drop. | ||||||||||||||||
| Comment by Ramon Orru [ 19/Feb/15 ] | ||||||||||||||||
|
We are experiencing this problem too. | ||||||||||||||||
| Comment by Christian Tonhäuser [ 27/Aug/12 ] | ||||||||||||||||
|
We seem to be running into the same issue. | ||||||||||||||||
| Comment by Scott Yancey [ 30/Jan/12 ] | ||||||||||||||||
|
FYI, your lists "Environment: OS X 64bit" but we're reproducing this on 64 bit CentOS. | ||||||||||||||||
| Comment by Daniel Crosta [ 30/Jan/12 ] | ||||||||||||||||
|
Additionally, sometimes dropDatabase will report success, but the database (and data files) still exist:
| ||||||||||||||||
| Comment by Daniel Crosta [ 30/Jan/12 ] | ||||||||||||||||
|
Trying to reproduce this again, this time I got a new error for step 5:
After which "show dbs" shows a partially-emptied db:
And in this case, I can verify that for 2 of the three shards, data files still exist:
The "blah2" database does not show up in config.databases, but "blah2.foo" shows up in config.collections with dropped == true. Further attempts to drop the database through mongos do not remove the spare data files nor change the output of "show dbs". I can, however, connect directly to the shards and drop the database there. This then causes the output of "show dbs" to become correct. | ||||||||||||||||
| Comment by Daniel Crosta [ 28/Jan/12 ] | ||||||||||||||||
|
(cleaned up formatting) |