[SERVER-31155] High memory & resources usage + very long boot time with many DBs Created: 19/Sep/17  Updated: 17/Oct/17  Resolved: 20/Sep/17

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: assaf-oren Assignee: Kelsey Schubert
Resolution: Duplicate Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 16.04


Issue Links:
Duplicate
duplicates SERVER-25025 Improve startup time when there are t... Closed
Participants:

 Description   

When using a lot of DBs (few thousands) mongod process take a lot of time to boot (minutes) and takes a lot of system resources afterwards.
Until boot is complete (minutes) the DB doesn't accept client connections.

It open all storage files (one per every collection and per index in every DB) + consumes a lot of memory. For 2000 DBs with 10 collections and 5 indexes per collection, there will be 100000 open file handlers.

On our use-case, we keep the DBs but access them rarely, usually only few DBs are relevant at any given moment.

I would expect mongo to lazy load the DBs when first connection is made.
(and hopefully unload if no active connection for a while)

Assaf



 Comments   
Comment by Kelsey Schubert [ 17/Oct/17 ]

Hi assaf-oren,

Sorry for the slow reply. Yes, it will close indexes or collections that have not been accessed recently.

Kind regards,
Kelsey

Comment by assaf-oren [ 21/Sep/17 ]

Thanks,
This configuration won't solve the very long boot time, but will save resources.
I'll try it.

Will it release all related resources for the collection, DB and indexes?

Comment by Kelsey Schubert [ 20/Sep/17 ]

Hi assaf-oren,

Thanks for the report. This appears to be a duplicate of the improvement request tracked in SERVER-25025. Please feel free to vote for it and watch it for updates.

By default, MongoDB will close inactive WiredTiger tables after 25 hours, however this can be tuned. From your description, you may see benefit using the following config, which reduces the idle time to 30 seconds:

storage:
    engine: wiredTiger
    wiredTiger:
        engineConfig:
            configString: "file_manager=(close_handle_minimum=100,close_idle_time=30,close_scan_interval=30)"

Kind regards,
Kelsey

Generated at Thu Feb 08 04:26:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.