[SERVER-3181] Add option to listDatabases to only get db names, not size info Created: 02/Jun/11  Updated: 15/Nov/21  Resolved: 20/Feb/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.2.13, 3.4.3, 3.5.4

Type: Improvement Priority: Major - P3
Reporter: Lei Zhang Assignee: Geert Bosch
Resolution: Done Votes: 10
Labels: PM248, bkp, commands, listDatabases, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux yq131 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
ext4 filesystem


Issue Links:
Backports
Depends
is depended on by SERVER-28924 Change databasesCloner to use the "na... Closed
is depended on by SERVER-28925 Use listDatabases({'nameOnly': true})... Closed
Documented
is documented by DOCS-10011 Need to document namesOnly option for... Closed
is documented by DOCS-10213 MongoDB 3.4 manual doesn't note `vers... Closed
Duplicate
is duplicated by SERVER-24948 initial sync failed because listDatab... Closed
is duplicated by SERVER-27666 listDatabases performance issue w/ re... Closed
is duplicated by SERVER-21066 Investigate listDatabases option to r... Closed
Related
related to DOCS-11104 Comment on: "manual/reference/command... Closed
related to SERVER-11051 listDatabases command reports existen... Closed
is related to TOOLS-1653 use the new higher performance listDa... Closed
is related to SERVER-6898 Allow all authenticated users to run ... Closed
is related to SERVER-21067 Investigate having the storage size o... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.4, v3.2
Sprint: Storage 2017-03-06
Participants:

 Description   

I have about 29377 files (including datafile and nsfile), 13666 databases. The total size is about 1TB. The mongod started with parameters "smallfiles" and "noprealloc".
When I run the command listDatabases, it takes too long on calculating "sizeOnDisk" and determining the empty of database, more than a hour sometimes. But I just need a list of database names. An option to not display size or empty probably makes sense.
I found that mongod will receive the listDatabases command in the following cases:
1. Dump all databases use mongodump
2. A Member of replica set doing initialSync
3. Add a new shard to a cluster.
In these cases, listDatabases may cause mongod overload and can not do the normal queries.



 Comments   
Comment by Githook User [ 19/Mar/17 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-3181 Add nameOnly option to listDatabases

(cherry picked from commit fff6d490ce43a6d3952cf2352f81c524b20e6f49)

Substantial changes were necessary for the backport, as the listDatabases command does not
implement filtering.

Conflicts:
jstests/concurrency/fsm_workloads/create_database.js
src/mongo/db/commands/list_databases.cpp
src/mongo/s/commands/cluster_list_databases_cmd.cpp
Branch: v3.2
https://github.com/mongodb/mongo/commit/d98f311b965505df9d7e6ff71084e8a2ad395f80

Comment by Githook User [ 03/Mar/17 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-3181 Add nameOnly option to listDatabases

(cherry picked from commit fff6d490ce43a6d3952cf2352f81c524b20e6f49)
Branch: v3.4
https://github.com/mongodb/mongo/commit/1349ae75590dee95475c5098ec6babce415ebfff

Comment by Eric Milkie [ 28/Feb/17 ]

It is currently scheduled to be backported to the 3.2 branch.

Comment by Chad Kreimendahl [ 28/Feb/17 ]

Will this be backported to 3.2, as requested?

Comment by Githook User [ 20/Feb/17 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-3181 Add nameOnly option to listDatabases
Branch: master
https://github.com/mongodb/mongo/commit/fff6d490ce43a6d3952cf2352f81c524b20e6f49

Comment by Chad Kreimendahl [ 19/Jan/17 ]

I'm thinking this happens on more than "initial sync", as we're seeing it happen 1x per replica member per minute.

Comment by Zhang Youdong [ 13/Jul/16 ]

continue the discuss on SERVER-24948

I create a pull request, see https://github.com/mongodb/mongo/pull/1100, and I will use it in our production environment.

Scott Hernandez I have rethink this change based on your tips.

  • Design the change
    • I think it shouldn't include the size info, to make it more understandable.
    • DBClient::getDatabaseNames return list<string>, so the change won't affect this interface.
    • I cannot think out any security related point.
    • It can be used from mongo shell, but I think it shouldn't set as default because many users may already use listDatabases to monitor the capacity of mongoDB.
  • Add sharding support
    • sharding support is also needed.
    • In sharding logic like getDBNamesListFromShard(), we can replace it with nameOnly version. etc
  • Research effect on driver/client (library)
    • It won't affect the driver, new drivers can use this new option.

In summary, my opinion is:

  • add an option to listDatabases command for mongod and mongos
  • use this option in the situation where only dbname is needed like initial sync.
Comment by Scott Hernandez (Inactive) [ 08/Jul/16 ]

Taking a look at this for initial sync improvements for 3.4

Generated at Thu Feb 08 03:02:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.