[SERVER-21402] Mongodb memory usage keep on increasing Created: 11/Nov/15  Updated: 25/Mar/19  Resolved: 14/Dec/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.0.6
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: sam anderson Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File db.serverStatus()     File ss.log.tar.gz     HTML File top    
Operating System: ALL
Participants:

 Description   

Configured Cache size: 20GB
MongoDB Ram usage: 114GB

MongoDB Ram utilization is keep on increasing and take all my ram (114 of 128) and it make Load average very high(sometimes above 250).

We need to restart mongod inorder to recover from there and within 2 days it takes all my ram.

I am using mongodb 3.0.6, WiredTiger with 1.5TB data and testing under High Read & Write application.
Server details Red Hat Enterprise Linux Server release 6.5
which is having 128GB Ram and 64 core processor.



 Comments   
Comment by Siddharth Shankar [ 25/Mar/19 ]

Hi,

 

Even in the current release of mongodb(4.0.4) still having the large number of open connection can utilise maximum if its memory.

Comment by Kelsey Schubert [ 14/Dec/15 ]

Hi samanderson@rishoncomm.in,

Sorry for the delay getting back to you. Each open connection and cursor increases the memory utilized by MongoDB.

The large number of connections and open cursors likely explains the behavior you are seeing.

For MongoDB-related support discussion please post on the mongodb-users group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

Kind regards,
Thomas

Comment by sam anderson [ 01/Dec/15 ]

Hi Alex,

objects : 4185991438
avgObjSize : 417.1831765815475
dataSize : 1746325205248
storageSize : 509287182336
numExtents : 0
indexes : 559
indexSize : 173359013888

Default connection size for our application is : 5000

Is the no. of connection affects memory?

Thanks
Sam

Comment by Alexander Gorrod [ 01/Dec/15 ]

Hi Sam,

From the statistics you uploaded, it appears as though you have a lot of open connections (2,500) and cursors (300+ thousand). It is likely that those connections and cursors are holding a lot of memory. Does your database have a lot of collections and/or indexes? Is your application opening lots of new connections to MongoDB?

Thanks,
Alex

Comment by sam anderson [ 26/Nov/15 ]

Hey!, any update....?

Comment by sam anderson [ 14/Nov/15 ]

ss.log

Comment by sam anderson [ 14/Nov/15 ]

Ramón thank u for ur reply.

I have Uploaded the ss.log file over here and https://10gen-httpsupload.s3.amazonaws.com/upload_forms/e412b029-5021-440b-81e4-12126af54974.html.

Is there anything u need?

Comment by Ramon Fernandez Marina [ 11/Nov/15 ]

samanderson, the wiredTigerCacheSizeGB parameter only limits the WiredTiger cache size, mongod needs additional memory to operate. From the information you uploaded I can see that there are not runaway cache issues:

                        "bytes currently in the cache" : 21225882917,
                        "maximum bytes configured"  : 21474836480,

In order to understand what's going on and whether the memory consumption you're seeing is expected or not we'll need additional information. Can you please run the following shell script on the affected node?

mongo --eval "while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:1}))); sleep(5000)}" >ss.log &

This script collects server status information every 5 seconds in a file called ss.log. Ideally it would be useful to collect data from a restart until memory grows past 100G for example. Once the data is collected please upload it here along with the server logs for the same period of time.

Thanks,
Ramón.

Generated at Thu Feb 08 03:57:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.