[SERVER-23362] Memory leak symptom in wiredtiger node with 1GB of memory Created: 26/Mar/16 Updated: 02/May/16 Resolved: 29/Mar/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tung Nguyen | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
Hi there, I noticed symptom which looks like memory leak in config servers of a sharded cluster (see atachment). If I restart mongod process of these config servers the memory utilization will go low, and then eventually go up until the process gets crashed or restarted manually.
Could you please advise? Thanks. |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 02/May/16 ] | |||||||||||||||||||||||||||||
|
Hi Jiri,
| |||||||||||||||||||||||||||||
| Comment by Jiri Nemecek [ 02/May/16 ] | |||||||||||||||||||||||||||||
|
Hello Ramon, can the --wiredTigerEngineConfigString="cache_size=200M" command line parameter be somehow put into the mongo conf file? We tried the storage.wiredTiger.engineConfig.cacheSizeGB as fractional number but that does not work as it is expecting an integer. Kind regards, | |||||||||||||||||||||||||||||
| Comment by Tung Nguyen [ 29/Mar/16 ] | |||||||||||||||||||||||||||||
|
Thank you all for your responses. | |||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 29/Mar/16 ] | |||||||||||||||||||||||||||||
|
tung@misfit.com, you're using machines with 1GB of memory, which is the minimum cache size WiredTiger will use. From the logs:
So the memory usage you're seeing is not a leak, but just increased cache usage until the server gets killed by the OOM killer. Please consider using machines with more memory; if that's not an option, you can consider lowering the cache size using the following command-line switch:
The above will configure the cache to be 200 megabytes. Since this is not a bug in the server, and the SERVER project is for reporting bugs or feature suggestions for the MongoDB server, I'm going to close this ticket. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources. Regards, | |||||||||||||||||||||||||||||
| Comment by Tung Nguyen [ 29/Mar/16 ] | |||||||||||||||||||||||||||||
|
Hi Bruce, My colleague found this on another config server: mongod process might be restarted when number of connections exceeds 60. I am not sure why it is 60, maybe it is because of the limited hardware (we are running t2.micro EC2 instances), or some other reason. And I found that not closing DB connection is actually a good practice so I do not think we have problem with connections staying there. But please help confirm if this is normal behavior? Anyway, this is the log of connections made, and at around 60, the number of connections went back to 1, 2, 3, ...
And from MongoDB Cloud Manager, we got this, which pretty much matches the time of logs above:
Thanks. | |||||||||||||||||||||||||||||
| Comment by Tung Nguyen [ 28/Mar/16 ] | |||||||||||||||||||||||||||||
|
Thanks Bruce for getting back! Please check new attachments. | |||||||||||||||||||||||||||||
| Comment by Bruce Lucas (Inactive) [ 28/Mar/16 ] | |||||||||||||||||||||||||||||
|
Hi Tung, Can you please archive (tar or zip) the $dbpath/diagnostic.data directory of a config server that experienced this problem, and attach to this ticket, together with the mongod log? This will contained detailed statistics about past memory usage that will help us understand the problem. Thanks, |