[SERVER-21837] MongoD memory usage higher than wiredTigerCacheSizeGB for primary in replica set Created: 10/Dec/15  Updated: 25/Jan/16  Resolved: 25/Jan/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nick Judson Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File memory-early.png     PNG File memory.png     File metrics.2015-12-16T05-49-34Z-00000    
Issue Links:
Duplicate
duplicates SERVER-20306 75% excess memory usage under WiredTi... Closed
Operating System: ALL
Steps To Reproduce:

3 node replica set on Windows (2 replicas and one arb).

Participants:

 Description   

I've been watching mongod memory fairly closely (SERVER-20991,
SERVER-20104 etc). With 3.2, memory usage on my non-replica-set deployment is excellent and it is well within expectations for the given wiredTigerCacheSizeGB.

Testing a replica-set with the same workload, the memory appears to push past this limit significantly: using a 4GB argument, actual memory usage is at 6.5 GB fairly quickly after kicking off the run.

I haven't done a full run yet, but will attempt to do so to see if memory keeps climbing or if it levels off.



 Comments   
Comment by Ramon Fernandez Marina [ 25/Jan/16 ]

Closing as a duplicate of SERVER-20306 after Bruce's analysis above.

Comment by Nick Judson [ 16/Dec/15 ]

Great - thanks for the analysis.

Comment by Bruce Lucas (Inactive) [ 16/Dec/15 ]

Thanks Nick. Here's what we see in the stats:

  • virtual memory fairly quickly climbs to about 6.5 GB and levels off
  • of that about 6 GB or so is in the tcmalloc heap, of which
    • the amount currently allocated ranges from 3-4 GB, mostly memory in the WT cache
    • the remaining 2-3 GB is in one of two tcmalloc free areas (central cache and pageheap)

Zooming in on an early part of the run:

  • up until about A we mostly see a well-behaved allocation arena: the heap size isn't much above the highwater mark of allocated memory, meaning we are effectively recycling freed memory.
  • starting around A, we see current allocated bytes remain about constant, because the cache is full, limiting the amount of memory that needs to be allocated.
  • but starting at about A, then especially at B, even though the amount of allocated memory is remaining constant, we see the tcmalloc heap size increase, which is accounted for by the free memory.
  • coinciding with this we see data being read into the cache.

This I believe is a sign of memory fragmentation: as memory is freed to evict data from the cache in order to make room to read new data in, the blocks that are freed aren't usable for the data being read in, so they remain on the free list and new memory is obtained. This appears to be to be an instance of SERVER-20306 (which is currently planned to be addressed in 3.4). In this case the data being read into the cache is for the oplog, which explains why you see this behavior with a replica set but not standalone.

Bruce

Comment by Nick Judson [ 16/Dec/15 ]

Here is the result of half of my test run. WT configured for 4GB, mongod > 6.

Comment by Bruce Lucas (Inactive) [ 10/Dec/15 ]

Hi Nick,

Can you please tar and attach to this ticket the $dbpath/diagnostic.data directory from the instance that showed the issue? This will contain data that will help us understand the reason for the excess memory usage.

Thanks,
Bruce

Comment by Nick Judson [ 10/Dec/15 ]

Hi Wan,

Imagine I'm provisioning a system which contains my software and Mongo (using WT). Let's say this system has 8G RAM and I need to leave 1GB for the OS, and 4GB for my application. This means I need to cap WT to around 3G. Running mongo as a stand-alone, I can do this, but running as part of a replica set, it appears I need to account for some extra usage. Is there a way to reasonably calculate this extra usage?

I'm not saying the --wiredTigerCacheSizeGB should account for everything (clearly it is WT only), but it would be nice to have a guide/docs for this scenario.

  • W2012
  • Measured in task mgr.

Nick.

Comment by Wan Bachtiar [ 10/Dec/15 ]

Hi Nick,

As noted in MongoDB documentation, the --wiredTigerCacheSizeGB only limits the size of the WiredTiger cache, not the total amount of memory used by mongod. The WiredTiger cache is only one component of the RAM used by MongoDB.

To better understand your test:

  • What is the specific Windows version ?
  • How do you measure the memory usage ? i.e. RAMMap, etc.

Kind regards,
Wan

Generated at Thu Feb 08 03:58:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.