[SERVER-10860] Mongos process is consuming more swap memory Created: 24/Sep/13  Updated: 11/Jul/16  Resolved: 05/Apr/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.4.1
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: Krishnachaitanya Thummuru Assignee: Unassigned
Resolution: Done Votes: 0
Labels: sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux CentOs 5.8


Attachments: File Logs.rar     Text File Replica_sets_stats.txt    
Operating System: ALL
Steps To Reproduce:

Provisioned 18 Million docs to database and each doc size around 1KB.

After 3 to 4 hours i observed swap consumption of the mongos process reached maximum swap.

Participants:

 Description   

Hi,

I am facing an issue related to the mongos process in my setup, mongos process is consuming more amount of swap memory, initially when I start the mongos process it consumes around 150-300MB of swap memory. After that i start provision data to the database, sometime later I observed mongos process is consumes around 4 GB of swap memory (swap memory which is configured to the Linux machine is 4.1GB), consumes almost all swap memory configured in setup.

Configuration of my setup is below.

  • Created 3 replica-sets which having 5 members each including arbiter
  • Created 3 config servers
  • Created the 32 mongos servers.

Each mongod and mongos processes are running in individual Linux machine.

Steps which i followed to recover swap

  • Stop mongos process
  • Start mongos process

After that consumption of swap memory reduce and consuming 150-300MB of swap.

Could you please suggest?

Thanks & Regards
Krishnachaitanya



 Comments   
Comment by Krishnachaitanya Thummuru [ 05/Apr/14 ]

Hi Thomas,

Thanks for update!

I had followed the all the steps recommend for the NUMA hardware.

Consumption of more swap memory of mongos process is resolved after upgrade to the 2.4.6, and as per Mike comments the mongod swap consumption fixed in the new build 2.4.7. for current even though mongod is taking more swap memory it not effecting much.

Please close the ticket.

Thanks for your support!

Thanks & Regards
Krishnachaitanya

Comment by Thomas Rueckstiess [ 04/Apr/14 ]

Hi Krishnachaitanya,

I noticed this issue is still unresolved and wanted to follow up with you. Have you confirmed that your NUMA is running with an interleave memory policy (i.e. --interleave=all) as per our recommendations for NUMA hardware? Is the swapping still an issue for you? Please let us know if we can close the ticket.

Regards,
Thomas

Comment by Krishnachaitanya Thummuru [ 11/Dec/13 ]

Hi Mike,

Thank you for the link
yes we are followed the steps mentioned in the document to create the setup and I verified the system parameter with the system parameters mentioned in the documentation and system parameters are looks similar as present in the documentation.

Will verify configuration one more time and let you know.

Thanks & Regards
Krishnachaitanya

Comment by Michael Grundy [ 09/Dec/13 ]

Hi Krishnachaitanya -

I should have thought of this earlier. You're running on NUMA hardware. Have you reviewed this documentation? http://docs.mongodb.org/manual/administration/production-notes/#mongodb-on-numa-hardware

Thanks!
Mike

Comment by Krishnachaitanya Thummuru [ 03/Dec/13 ]

Hi Mike,

Please find my inputs for your queries
1. Current readahead configuration in machine

  1. blockdev --getra /dev/mapper/VolGroup00-LogVol03
    256
    #

If i understood correctly the current configuration of the readahead is set to 256 on setup even if the mongod uses 1K the rest of the 127K staying in system cache and it is not reverting back right? To avoid the remainder will be in system cache i need to reduce the readahead from 256 to 32 ?

2. Kernel version

  1. cat /etc/redhat-release
    CentOS release 5.7 (Final)
    #
  1. uname -r
    2.6.18-274.18.1.el5
    #

3. Hardware
I am using UCS-B hardware and i created the virtual machine on top of the blades

Thanks & Regards
Krishnachaitanya

Comment by Michael Grundy [ 02/Dec/13 ]

Hi Krishnachaitanya -

First, I meant to say 32 If the volume is being shared by multiple databases, then tuning is a trade off between the different databases' characteristics, and other programs accessing the volume. With the default (and I'm guessing sda2 from the info you sent previously) of 256 and a block size of 512, each read will pull in 128K. If your document size is about 1K, then mongod will use the 1K, but the remainder will be in system cache. A readahead of 32 will only pull in 16K.

Second, the swap numbers for mongod are concerning, more than from a "general performance improvement that setting readahead properly will give" point of view. Since the numbers you show from top in a previous comment don't show out of control mongod resident size, the swapped numer seems excessive.

What Distro / Kernel version / Hardware are you running on ?

Thanks!
Mike

Comment by Krishnachaitanya Thummuru [ 02/Dec/13 ]

Hi Mike,

After upgrade to 2.4.6 mongos process consumption of swap in between 316MB to 515MB and it's stayed there, and upgrade to 2.4.6 is helped for me.

Thanks for your update, If i set the readahead value to 30, will it impact any other document which loads more than 30 in the same machine?
How can i calculate the readahead value based on the document size?

Currently testing is in progress with mongo 2.4.6 and so for it's working fine.
To test your suggestions related to mongod swap consumption i can do after completion of the current testing only.

As per your earlier comments if the fix for mongod swap consumption provide in the new build 2.4.7 then it's good.

Thanks & Regards
Krishnachaitanya

Comment by Michael Grundy [ 02/Dec/13 ]

Hi Krishnachaitanya -

Did your mongos swap behavior stay reasonable after the upgrade? If you are still observing swap growth on mongos can you upload /proc/<pid>/smaps for the mongos process please?

For your mongod memory consumption, you should drop your readahead to a lower value (blockdev --setra) on the volume containing your dbpath. 30 would probably be a more reasonable generic value, you can adjust it based on your document size to prevent loading unnecessary blocks into cache when reading in documents.

Thanks!
Mike

Comment by Krishnachaitanya Thummuru [ 30/Sep/13 ]

Any update ?
Could you please let me know.

Thanks & Regards
Krishnachaitanya

Comment by Krishnachaitanya Thummuru [ 26/Sep/13 ]

Hi Mike,

Please find updated stat file attached.

As you mentioned earlier 2.4.7 have fixes related to high memory usage of the mongod process, right now i am testing for the production 2.4.6 and i can't test the 2.4.7 before actual release. Will test 2.4.7 once i done testing with 2.4.6.

Thanks for suggesting me and giving latest build.

Without upgrade to 2.4.7 if you could give me any suggestion for reduce the swap memory usage of mongod process that would be helpful for me.

Mongos process consumption:
I had provisioned the 4 Million documents to the DB for testing mongos process swap consumption, mongos process consumption of swap memory looks okay to me.

After upgrade When i am testing the mongos process consumption of swap is increased from 316MB to 515MB and it's stayed there, it seems to upgrade to 2.4.6 is helped for me.

FYI
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
20287 root 15 0 542m 27m 3656 S 2.3 0.3 21:22.21 515m mongos
4067 root 15 0 159m 6952 1360 S 0.0 0.1 2:21.19 152m snmpd

Thanks & Regards
Krishnachaitanya

Comment by Krishnachaitanya Thummuru [ 26/Sep/13 ]

Stats of the replica-sets

Comment by Michael Grundy [ 25/Sep/13 ]

Could you also upload the contents of /proc/<pid>/stat (for the same mongod processes) and the output of free for each node?

Thanks!
Mike

Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ]

System logs and mongostat logs

Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ]

Hi Mike,

Thanks for your inputs!

Sorry for not to mentioned the mongod process issue earlier, i mainly concentrated on the mongos process because I am using for provisioning due to this reason i did not mentioned it.

largest mongod process is involved in the provisioning and same amount of swap is used by the all mongod process which is involved in provisioning, the smaller mongod process are different replica-set members which are in same nodes but involved in other tasks, I am interested on mongod process which is using more swap.

Please find the attachment for logs which you are asking me.
Logs are related to the mongod process which are using high swap.

Thanks & Regards
Krishnachaitanya

Comment by Michael Grundy [ 25/Sep/13 ]

Hi Krishnachaitanya -

First, as to your question about metadata upgrade, that only applies to a 2.2 -> 2.4 upgrade, not to a 2.4.x to 2.4.x

You hadn't mentioned mongod having high swap, there are some fixes going into 2.4.7 that address high memory usage in mongod (and if you want to give it a test drive before it's released you can get it at http://downloads.mongodb.org/linux/mongodb-linux-x86_64-v2.4-latest.tgz).

That said, I would look to system configuration issues first. The information you posted shows ~94 gig of memory, but the largest mongod (are these all on the same machine, or sampled from different nodes?) is only 25g total and there is 65g free.

Check the following:

  • cat /proc/sys/vm/swappiness (the default is 60)
  • run mongostat --all , look at the mapped, vsize, res, and non-mapped values
  • run sudo blockdev --report , high readahead values affect mongod's ability to utilize as much memory as it could (read http://www.kchodorow.com/blog/2012/05/10/thursday-5-diagnosing-high-readahead/ )
  • post the output of all of these and /proc/<pid>/status where <pid> is the pid of the mongod process

Thanks
Mike

Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ]

Hi

After upgrade memory consumption of the mongod replica set member below.

top - 00:41:14 up 1 day, 22:26, 1 user, load average: 1.50, 3.21, 3.43
Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8%us, 0.5%sy, 0.0%ni, 96.8%id, 1.2%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 99003004k total, 30394716k used, 68608288k free, 1630408k buffers
Swap: 6127608k total, 0k used, 6127608k free, 28167628k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
31891 root 15 0 25.4g 758m 696m S 1.0 0.8 6:41.23 24g mongod
14426 root 15 0 6937m 2.9g 2.9g S 0.3 3.1 9:47.63 3.8g mongod
31679 root 15 0 3766m 61m 12m S 0.7 0.1 13:31.46 3.6g mongod
32024 root 15 0 3525m 75m 30m S 0.7 0.1 6:41.65 3.4g mongod
4521 root 15 0 176m 29m 4360 S 0.0 0.0 0:32.35 146m snmpd

In all members the mongod process swap memory consumption is same.

The memory consumption of the mongos process is not yet tested, will let you know as early as possible

I am not sure why mongod consuming more amount of the swap, could you please give me any suggestion?

Thanks & Regards
Krishnachaitanya

Comment by Krishnachaitanya Thummuru [ 25/Sep/13 ]

Hi

Upgraded a Replica Set from MongoDB 2.4.1 to MongoDB 2.4.6.

Procedure to upgraded the shard cluster "Meta-Data Upgrade Procedure" steps mentioned in following link is required for 2.4.1 to 2.4.6?
http://docs.mongodb.org/manual/release-notes/2.4-upgrade/

Please suggest

Thanks & Regards
Krishnahchaitanya

Comment by Krishnachaitanya Thummuru [ 24/Sep/13 ]

Thanks for your update!

Will upgrade to 2.4.6 and will let you know

Thanks & Regards
Krishnachaitanya

Comment by Michael Grundy [ 24/Sep/13 ]

Upgrading to the latest version of MongoDB (currently 2.4.6) should resolve the memory leaks you are seeing. There are also several other important bug and stability fixes that have been included since 2.4.1, so I strongly recommend upgrading at your earliest convenience. As part of the 2.4 maintenance series, it should be a drop-in replacement. Please let us know when you have had a chance to upgrade and confirm that the upgrade fixes the issue you were seeing.

Thanks!
Mike

Generated at Thu Feb 08 03:24:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.