[SERVER-27919] when too many connections to mongod server, too much heap memory free and keep in physical memory which don't be released Created: 06/Feb/17  Updated: 10/Oct/17  Resolved: 07/Feb/17

Status: Closed
Project: Core Server
Component/s: Internal Code
Affects Version/s: 3.2.8
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: shawn Assignee: Mark Agarunov
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2017-02-07 at 4.06.54 PM.png     Zip Archive diagnostic.data.zip    
Operating System: ALL
Participants:

 Description   

config:
wiredTiger Cache size : 2GB
max connections: 3000
mongod Version: 3.2.8
total dataset size: 16GB

how to find this question:
we set max connections as 3000, so when connections reach to 3000, we received sms warning. login to server, found that: this mongod use 39GB physical memory, as follows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND                                                                                                        
13154 mongod  20   0 46.9g  39g 9240 S  2.0 62.0   3815:20 mongod

connecto to mongod , seeing from serverStatus:

 "generic" : {
                "current_allocated_bytes" : NumberLong("11833558352"),
                "heap_size" : NumberLong("48540131328")
        },
        "tcmalloc" : {
                "pageheap_free_bytes" : NumberLong("31037857792"),
                "pageheap_unmapped_bytes" : NumberLong("5174722560"),
                "max_total_thread_cache_bytes" : NumberLong(1073741824),
                "current_total_thread_cache_bytes" : 384155856,
                "central_cache_free_bytes" : 90345984,
                "transfer_cache_free_bytes" : 19490784,
                "thread_cache_free_bytes" : 384155856,
                "aggressive_memory_decommit" : 0
        }

but, pageheap_free_bytes not be released.

Once the connecitons increases, the memory use increases too.

monitor data from my data collector:



 Comments   
Comment by Mark Agarunov [ 07/Feb/17 ]

Hello shawn001,

Thank you for providing the diagnostic data. Looking over this, it seems that the behavior you're describing is due to the kernel's memory model. In the following screenshot:

At point A the memory is allocated for an operation, immediately after, once the operation has completed and the memory is no longer needed, the majority of that allocated memory is considered free, as can be seen in the pageheap_free_bytes in the above screenshot. However, as you note it is still considered as memory used by the monogod process. This is due to how the kernel allocates memory. Even if the memory is freed, the kernel will hold it allocated in case it is used again, to skip having to reallocate that memory, unless there is memory pressure. That is, until something else requests the memory, the kernel will generally keep the memory "used".

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Thanks,
Mark

Comment by shawn [ 07/Feb/17 ]

@Thomas Schubert

my diagnostic.data as follows.

diagnostic.data.zip

thanks.

Comment by Kelsey Schubert [ 06/Feb/17 ]

Hi shawn001,

Thanks for reporting this behavior. So we can investigate, would you please attach an archive of the diagnostic.data directory for the affected mongod?

Regards,
Thomas

Comment by shawn [ 06/Feb/17 ]

os version: CentOS release 6.5 (Final)

Generated at Thu Feb 08 04:16:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.