[SERVER-10860] Mongos process is consuming more swap memory Created: 24/Sep/13 Updated: 11/Jul/16 Resolved: 05/Apr/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.4.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Krishnachaitanya Thummuru | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | sharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux CentOs 5.8 |
||
| Attachments: |
|
| Operating System: | ALL |
| Steps To Reproduce: | Provisioned 18 Million docs to database and each doc size around 1KB. After 3 to 4 hours i observed swap consumption of the mongos process reached maximum swap. |
| Participants: |
| Description |
|
Hi, I am facing an issue related to the mongos process in my setup, mongos process is consuming more amount of swap memory, initially when I start the mongos process it consumes around 150-300MB of swap memory. After that i start provision data to the database, sometime later I observed mongos process is consumes around 4 GB of swap memory (swap memory which is configured to the Linux machine is 4.1GB), consumes almost all swap memory configured in setup. Configuration of my setup is below.
Each mongod and mongos processes are running in individual Linux machine. Steps which i followed to recover swap
After that consumption of swap memory reduce and consuming 150-300MB of swap. Could you please suggest? Thanks & Regards |
| Comments |
| Comment by Krishnachaitanya Thummuru [ 05/Apr/14 ] |
|
Hi Thomas, Thanks for update! I had followed the all the steps recommend for the NUMA hardware. Consumption of more swap memory of mongos process is resolved after upgrade to the 2.4.6, and as per Mike comments the mongod swap consumption fixed in the new build 2.4.7. for current even though mongod is taking more swap memory it not effecting much. Please close the ticket. Thanks for your support! Thanks & Regards |
| Comment by Thomas Rueckstiess [ 04/Apr/14 ] |
|
Hi Krishnachaitanya, I noticed this issue is still unresolved and wanted to follow up with you. Have you confirmed that your NUMA is running with an interleave memory policy (i.e. --interleave=all) as per our recommendations for NUMA hardware? Is the swapping still an issue for you? Please let us know if we can close the ticket. Regards, |
| Comment by Krishnachaitanya Thummuru [ 11/Dec/13 ] |
|
Hi Mike, Thank you for the link Will verify configuration one more time and let you know. Thanks & Regards |
| Comment by Michael Grundy [ 09/Dec/13 ] |
|
Hi Krishnachaitanya - I should have thought of this earlier. You're running on NUMA hardware. Have you reviewed this documentation? http://docs.mongodb.org/manual/administration/production-notes/#mongodb-on-numa-hardware Thanks! |
| Comment by Krishnachaitanya Thummuru [ 03/Dec/13 ] |
|
Hi Mike, Please find my inputs for your queries
If i understood correctly the current configuration of the readahead is set to 256 on setup even if the mongod uses 1K the rest of the 127K staying in system cache and it is not reverting back right? To avoid the remainder will be in system cache i need to reduce the readahead from 256 to 32 ? 2. Kernel version
3. Hardware Thanks & Regards |
| Comment by Michael Grundy [ 02/Dec/13 ] |
|
Hi Krishnachaitanya - First, I meant to say 32 Second, the swap numbers for mongod are concerning, more than from a "general performance improvement that setting readahead properly will give" point of view. Since the numbers you show from top in a previous comment don't show out of control mongod resident size, the swapped numer seems excessive. What Distro / Kernel version / Hardware are you running on ? Thanks! |
| Comment by Krishnachaitanya Thummuru [ 02/Dec/13 ] |
|
Hi Mike, After upgrade to 2.4.6 mongos process consumption of swap in between 316MB to 515MB and it's stayed there, and upgrade to 2.4.6 is helped for me. Thanks for your update, If i set the readahead value to 30, will it impact any other document which loads more than 30 in the same machine? Currently testing is in progress with mongo 2.4.6 and so for it's working fine. As per your earlier comments if the fix for mongod swap consumption provide in the new build 2.4.7 then it's good. Thanks & Regards |
| Comment by Michael Grundy [ 02/Dec/13 ] |
|
Hi Krishnachaitanya - Did your mongos swap behavior stay reasonable after the upgrade? If you are still observing swap growth on mongos can you upload /proc/<pid>/smaps for the mongos process please? For your mongod memory consumption, you should drop your readahead to a lower value (blockdev --setra) on the volume containing your dbpath. 30 would probably be a more reasonable generic value, you can adjust it based on your document size to prevent loading unnecessary blocks into cache when reading in documents. Thanks! |
| Comment by Krishnachaitanya Thummuru [ 30/Sep/13 ] |
|
Any update ? Thanks & Regards |
| Comment by Krishnachaitanya Thummuru [ 26/Sep/13 ] |
|
Hi Mike, Please find updated stat file attached. As you mentioned earlier 2.4.7 have fixes related to high memory usage of the mongod process, right now i am testing for the production 2.4.6 and i can't test the 2.4.7 before actual release. Will test 2.4.7 once i done testing with 2.4.6. Thanks for suggesting me and giving latest build. Without upgrade to 2.4.7 if you could give me any suggestion for reduce the swap memory usage of mongod process that would be helpful for me. Mongos process consumption: After upgrade When i am testing the mongos process consumption of swap is increased from 316MB to 515MB and it's stayed there, it seems to upgrade to 2.4.6 is helped for me. FYI Thanks & Regards |
| Comment by Krishnachaitanya Thummuru [ 26/Sep/13 ] |
|
Stats of the replica-sets |
| Comment by Michael Grundy [ 25/Sep/13 ] |
|
Could you also upload the contents of /proc/<pid>/stat (for the same mongod processes) and the output of free for each node? Thanks! |
| Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ] |
|
System logs and mongostat logs |
| Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ] |
|
Hi Mike, Thanks for your inputs! Sorry for not to mentioned the mongod process issue earlier, i mainly concentrated on the mongos process because I am using for provisioning due to this reason i did not mentioned it. largest mongod process is involved in the provisioning and same amount of swap is used by the all mongod process which is involved in provisioning, the smaller mongod process are different replica-set members which are in same nodes but involved in other tasks, I am interested on mongod process which is using more swap. Please find the attachment for logs which you are asking me. Thanks & Regards |
| Comment by Michael Grundy [ 25/Sep/13 ] |
|
Hi Krishnachaitanya - First, as to your question about metadata upgrade, that only applies to a 2.2 -> 2.4 upgrade, not to a 2.4.x to 2.4.x You hadn't mentioned mongod having high swap, there are some fixes going into 2.4.7 that address high memory usage in mongod (and if you want to give it a test drive before it's released you can get it at http://downloads.mongodb.org/linux/mongodb-linux-x86_64-v2.4-latest.tgz). That said, I would look to system configuration issues first. The information you posted shows ~94 gig of memory, but the largest mongod (are these all on the same machine, or sampled from different nodes?) is only 25g total and there is 65g free. Check the following:
Thanks |
| Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ] |
|
Hi After upgrade memory consumption of the mongod replica set member below. top - 00:41:14 up 1 day, 22:26, 1 user, load average: 1.50, 3.21, 3.43 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND In all members the mongod process swap memory consumption is same. The memory consumption of the mongos process is not yet tested, will let you know as early as possible I am not sure why mongod consuming more amount of the swap, could you please give me any suggestion? Thanks & Regards |
| Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ] |
|
Hi Upgraded a Replica Set from MongoDB 2.4.1 to MongoDB 2.4.6. Procedure to upgraded the shard cluster "Meta-Data Upgrade Procedure" steps mentioned in following link is required for 2.4.1 to 2.4.6? Please suggest Thanks & Regards |
| Comment by Krishnachaitanya Thummuru [ 24/Sep/13 ] |
|
Thanks for your update! Will upgrade to 2.4.6 and will let you know Thanks & Regards |
| Comment by Michael Grundy [ 24/Sep/13 ] |
|
Upgrading to the latest version of MongoDB (currently 2.4.6) should resolve the memory leaks you are seeing. There are also several other important bug and stability fixes that have been included since 2.4.1, so I strongly recommend upgrading at your earliest convenience. As part of the 2.4 maintenance series, it should be a drop-in replacement. Please let us know when you have had a chance to upgrade and confirm that the upgrade fixes the issue you were seeing. Thanks! |