[SERVER-23034] Large number of collections causes extreme dropDatabase slowness Created: 08/Mar/16 Updated: 09/Jun/16 Resolved: 11/Apr/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Justin Lintz | Assignee: | Kelsey Schubert |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 12.04 Linux Set of 3 in a replicaset |
||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
Create a database with few thousand collections, try and drop the database. For a database of only a few hundred megabytes, took over a half hour to delete. "show dbs" was hung during that time as well. If I just create a database with one collection and one document, I can delete things just fine. I'm currently graphing all metrics from serverStatus so let me know if theres any values you think are relevant to help debugging the issue. I'll add more information to this issue as I find it since I'm still running tests but wanted to get the discussions going. |
| Comments |
| Comment by Kelsey Schubert [ 11/Apr/16 ] | ||||||||||||||||||||||
|
Hi jlintz, We haven’t heard back from you for some time, so I’m Kind regards, | ||||||||||||||||||||||
| Comment by Kelsey Schubert [ 18/Mar/16 ] | ||||||||||||||||||||||
|
Hi jlintz, Thank you for providing a repro script. Unfortunately, I have not been able to reproduce this issue: I increased the number of collections in your repro script to 25,000 and observed that the dropDatabase command took about 30 seconds to complete.
To continue to investigate this issue, can you please execute the following shell script before dropping the database:
This will collect iostat data each second, and will help us understand what is going on. After dropping the database, please upload both the iostat.log and diagnostic.data to this ticket. Thank you, | ||||||||||||||||||||||
| Comment by Justin Lintz [ 11/Mar/16 ] | ||||||||||||||||||||||
|
Just confirmed the issue is present as a standalone instance as well | ||||||||||||||||||||||
| Comment by Justin Lintz [ 11/Mar/16 ] | ||||||||||||||||||||||
|
I've created a javascript file that re-create's the issue. I've attached the JS file, diagnostics directory and a copy of the config file to this issue and a couple relevant log lines. So far I've only tested in a replica set of 3 machines, I'll try and test on a standalone instance and get back to you | ||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 10/Mar/16 ] | ||||||||||||||||||||||
|
jlintz, can you please upload the contents of the diagnostic.data directory from your dbpath? Can you also provide details on the size of those collections and the average amount of indexes in them? Any other information you can provide that may help us reproduce this issue on our end will be helpful. Thanks, | ||||||||||||||||||||||
| Comment by Susan LoVerso [ 09/Mar/16 ] | ||||||||||||||||||||||
|
Hello jlintz. Could you create a js script that shows the problem? Does this happen also on a single standalone mongod instead of with replication? It would be helpful to reproduce it here. Something that creates some number of databases, then creates the large number of collections in each database, inserts the data and then does the drop. Also, if you want to attach a tar of the serverStatus FTDC data, we could look at that. | ||||||||||||||||||||||
| Comment by Justin Lintz [ 08/Mar/16 ] | ||||||||||||||||||||||
|
Also should note, there is a ton of disk write activity during the drop, around 125 MB/sec or more of constant writes |