[SERVER-17078] show databases taking extraordinarily long with wiredTiger Created: 27/Jan/15 Updated: 18/Jul/16 Resolved: 26/Jun/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Performance, WiredTiger |
| Affects Version/s: | 3.0.0-rc6 |
| Fix Version/s: | 3.0.5, 3.1.5 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Chad Kreimendahl | Assignee: | Michael Cahill (Inactive) |
| Resolution: | Done | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Backport Completed: | |||||||||||||||||||||||||
| Steps To Reproduce: | "show databases" or doing a listdatabases command through the api/C#/java/etc. |
||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||||||||||
| Description |
| Comments |
| Comment by Githook User [ 10/Jul/15 ] | |||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: (cherry picked from commit c7bab6f8cfbd610ff1690cefaccb6fc83658f517) | |||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 29/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: (cherry picked from commit b956408d23d6abe21ce3f833c92d3ee011cfc845) | |||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 29/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: (cherry picked from commit 74cea8701c115b9327523691659ed68c86c1f652) | |||||||||||||||||||||||||||||||||||||
| Comment by Michael Cahill (Inactive) [ 15/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
paul.reed, the fast "size" mode is enabled by default for listDatabases with WiredTiger in MongoDB master here: https://github.com/mongodb/mongo/commit/c7bab6f8cfbd610ff1690cefaccb6fc83658f517 A backport to the 3.0 branch is in progress. If you are requesting that listDatabases not return size information at all, can you please vote for | |||||||||||||||||||||||||||||||||||||
| Comment by Paul Reed [ 12/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
Will "Size" become the default option and would really like the ability to be able listDatabases - without size requirements - this becoming the fastest method. | |||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 10/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: Add a "statistics=(size)" mode to statistics cursors that just gets the filesize without opening anything. refs | |||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 10/Jun/15 ] | |||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'}Message: | |||||||||||||||||||||||||||||||||||||
| Comment by Zhou Jiaqing [ 15/May/15 ] | |||||||||||||||||||||||||||||||||||||
|
This issue blocked our upgrade process. Hope someone will fix this problem soon. wt engine is really attractive to us! | |||||||||||||||||||||||||||||||||||||
| Comment by Paul Reed [ 13/May/15 ] | |||||||||||||||||||||||||||||||||||||
|
This is a major issue for me. | |||||||||||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 14/Apr/15 ] | |||||||||||||||||||||||||||||||||||||
|
Hi João, I suspect that the issue with creating a new database is not connected to this ticket. I created 1000 empty databases, with no significant performance degradation between creating the first db and creating the 1000th. So it seems that the simple number of dbs is not an issue with creating a new one, and that distinguishes it from the issue on this ticket, which manifests simply due to the number of dbs. So please open a new ticket as you suggested. While doing so, would you be able to reproduce the problem while runnng the following data collection script, which collects serverStatus information once per second:
Please reproduce the problem as follows:
When you open the new ticket, please attach ss.log, the mongod log file covering the repro, and the timeline. Thanks, | |||||||||||||||||||||||||||||||||||||
| Comment by João Soares [ 14/Apr/15 ] | |||||||||||||||||||||||||||||||||||||
|
Hi, we are using multi tenancy on the database level, so each of our clients get its own database. We currently have more than 1 thousand databases and issuing
takes us several minutes to get a result. We are also experiencing some really slow performance on a first write to a new database. Steps to reproduce:
Does anyone know if these two problems are somehow related? I haven't found any ticket related to my second issue but I can open one... | |||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 23/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
WIth the understanding that this can be improved substantially, you should see that subsequent calls to listDatabases are much faster. | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 23/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
This is specifically making any call to listDatabases take extraordinarily long. This makes "DatabaseExists()" calls in C# quite long. | |||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 09/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
The size on disk can be quite different than the size of the documents stored in-memory. In mmap because of record padding and in wired tiger because of compression. I would venture to say that the size is not "necessary" – it's an interface that was designed when getting the value was extremely cheap. | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 09/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
Is it possible to get the in-memory size vs on-disk size? Would they correlate well? Is size necessary when listing databases? | |||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 08/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
We have a fix in WT which makes this about 50% faster, but that's not really enough. | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 05/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
Thanks for the update. For us this causes substantial issues when our code migration tool runs our "database upgrade" functions, which list each database, open them and see what work needs to be done for when we deploy new code. Ultimately we only do that about once a month, so it's not terrible. The UI tools we use for manually editing on our own systems, including dev take a long time to open because of this... however, it's not a show stopper, as it doesn't get called when using our software. | |||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 05/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
Work is in progress on this, but not leaving this in as a showstopper. We'll evaluate solutions when they are prepared. | |||||||||||||||||||||||||||||||||||||
| Comment by Mark Benvenuto [ 04/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
Basically, the issue is that we have two O(N) loops where N = 2 * Collections + User Indexes that hurt performance during the statistics scan. Repro script
VS Profiler Trace
The problem is this code in __wt_open and __wt_block_open. This is where ~30% of the time is spent just checking the stats of the collection in this particular stack trace.
| |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 03/Feb/15 ] | |||||||||||||||||||||||||||||||||||||
|
Anything? | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 28/Jan/15 ] | |||||||||||||||||||||||||||||||||||||
|
Here's the output from a process monitor of what happens with mongod when I run show databases. | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 28/Jan/15 ] | |||||||||||||||||||||||||||||||||||||
|
Sure. Of note: this does run quite a bit faster on a system with SSDs, but it's still 10 seconds, while fully pegging CPU. Also, when getting the stats on databases for you, I noticed that db.stats() takes quite a bit of time. It seems to correlate fairly well with the number of indexes in a given database, versus size.
| |||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 28/Jan/15 ] | |||||||||||||||||||||||||||||||||||||
|
sallgeud, can you please provide some more details about your setup and what kind of load it has? I launched both a stand-alone node and a 3-node replica set (on the same machine) and fired up 700 processes inserting data in 30 databases, each with 10 collections. The load on my system is past 300:
but even under these conditions running the "show databases" command takes only a fraction of a second. | |||||||||||||||||||||||||||||||||||||
| Comment by Chad Kreimendahl [ 27/Jan/15 ] | |||||||||||||||||||||||||||||||||||||
|
Of note is that it also appears to be using the entirety of a core of a CPU, no real change in memory usage and no substantial io load on the disks. |