<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:52:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-19795] mongod memory consumption higher than WT cache size (at least on Windows 2008 R2)</title>
                <link>https://jira.mongodb.org/browse/SERVER-19795</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Although mongod.exe process memory is well capped ,as required by the --wiredTigerCacheSizeGB, the OS (Windows 2008 R2) keeps huge part of os memory cache active on files.&lt;/p&gt;

&lt;p&gt;See screen shot of RMAP tool : &lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;green : no problem because of use of --wiredTigerEngineConfigString direct_io=&lt;span class=&quot;error&quot;&gt;&amp;#91;data&amp;#93;&lt;/span&gt; , no data file part is cached by the os, no problem.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;red : the os does keep in ram large parts (see column &quot;Active File&quot;), this is ok , but the issue here is that the sum of ram used because of mongod ( &quot;mongod privateByte&quot; + &quot;os active file&quot; ) exceed --wiredTigerCacheSizeGB , in fact at then end it would have the whole size of the database in memory. And this would lead to OOM we cannot prevent since we cannot cap &quot;OS active file&quot; locked by the OS as asked by WT.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;the &quot;--wiredTigerEngineConfigString direct_io=&lt;span class=&quot;error&quot;&gt;&amp;#91;data&amp;#93;&lt;/span&gt;&quot; is a workaround to this problem , but then it lead to too much slowness for queries and in fact is not applicable for our volumes of process and data.&lt;/p&gt;

&lt;p&gt;Could you make --wiredTigerCacheSizeGB take in acount the whole ram used because of mongo (os+process) , please ?&lt;/p&gt;


&lt;p&gt;Mongo64_2008+\mongod.exe&quot; --port 4444 --dbpath D:\Homeware\XOne_services\PreBETA\MongoData\CurvesFR --directoryperdb --journal --nohttpinterface --wiredTigerCacheSizeGB 1 --wiredTigerDirectoryForIndexes --replSet MongoServiceCacheCurves --oplogSize 1024 --storageEngine wiredTiger --auth --keyFile x.keyfile&lt;/p&gt;</description>
                <environment>W 2008 R2</environment>
        <key id="224389">SERVER-19795</key>
            <summary>mongod memory consumption higher than WT cache size (at least on Windows 2008 R2)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ramon.fernandez@mongodb.com">Ramon Fernandez Marina</assignee>
                                    <reporter username="marc.girollet@sgcib.com">Marc Girollet</reporter>
                        <labels>
                            <label>RF</label>
                            <label>WTmem</label>
                    </labels>
                <created>Thu, 6 Aug 2015 09:57:21 +0000</created>
                <updated>Thu, 11 Jan 2018 15:24:36 +0000</updated>
                            <resolved>Sat, 26 Sep 2015 12:12:15 +0000</resolved>
                                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="1063833" author="nick@innsenroute.com" created="Mon, 19 Oct 2015 03:13:10 +0000"  >&lt;p&gt;@Michael Cahill - I found this an interesting read: &lt;a href=&quot;http://winntfs.com/2012/11/29/windows-write-caching-part-2-an-overview-for-application-developers/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://winntfs.com/2012/11/29/windows-write-caching-part-2-an-overview-for-application-developers/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;...&quot;Some well known applications such as Microsoft SQL and the Microsoft JET database (ships with Windows Server SKUs) specify FILE_FLAG_NO_BUFFERING with the CreateFile API.&quot;...&lt;/p&gt;

&lt;p&gt;Is there a flag where we can get verbose output from WT (and track the frequency of flushes etc.)?&lt;/p&gt;</comment>
                            <comment id="1053970" author="nick@innsenroute.com" created="Wed, 7 Oct 2015 15:41:05 +0000"  >&lt;p&gt;Thanks for looking @Michael Cahill - appreciated. I don&apos;t know if/what the fix might be and unfortunately I don&apos;t have a C++ build environment. Looking at the windows docs and experimenting with artificially restricted system caches doesn&apos;t reveal much of interest - other than for my workload it doesn&apos;t appear to provide any benefit.&lt;/p&gt;

&lt;p&gt;I don&apos;t see how it&apos;s possible to restrict the system file cache by process, so it may be out of your hands. Any in-house Windows experts able to comment?&lt;/p&gt;

&lt;p&gt;The part I find odd is that the cache never seems to be released. I&apos;m wondering if there is something non-standard about the way WT creates/locks files.&lt;/p&gt;</comment>
                            <comment id="1052023" author="michael.cahill" created="Tue, 6 Oct 2015 02:19:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nick%40innsenroute.com&quot; class=&quot;user-hover&quot; rel=&quot;nick@innsenroute.com&quot;&gt;nick@innsenroute.com&lt;/a&gt;, the &lt;tt&gt;os_cache_max&lt;/tt&gt; setting in WiredTiger currently relies on &lt;tt&gt;posix_fadvise&lt;/tt&gt;: we don&apos;t have a Windows implementation so it will not have any effect.&lt;/p&gt;

&lt;p&gt;I&apos;m genuinely sorry that you are having problems using WiredTiger on Windows.  It is a good sign that enabling direct I/O helps in some cases: what I suspect that relies on is the I/O subsystem being fast enough that reads / writes don&apos;t stall by going to disk synchronously.&lt;/p&gt;

&lt;p&gt;I did a review today of what interfaces are available on Windows and how WiredTiger uses them.  One thing I noticed is that WiredTiger&apos;s &lt;tt&gt;direct_io&lt;/tt&gt; setting does two things on Windows: it sets &lt;tt&gt;FILE_FLAG_NO_BUFFERING&lt;/tt&gt; to disable filesystem cache and &lt;tt&gt;FILE_FLAG_WRITE_THROUGH&lt;/tt&gt; to make writes synchronous to disk.&lt;/p&gt;

&lt;p&gt;I don&apos;t think WiredTiger needs &lt;tt&gt;FILE_FLAG_WRITE_THROUGH&lt;/tt&gt; for correctness: we will &lt;b&gt;also&lt;/b&gt; call &lt;tt&gt;FlushFileBuffers&lt;/tt&gt; for durability.  Given that, it may be worth trying performance runs without writethrough semantics.  Unfortunately, doing that requires a source code change:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;diff --git a/src/third_party/wiredtiger/src/os_win/os_open.c b/src/third_party/wiredtiger/src/os_win/os_open.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;index c7b3040..633db36 100644&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;--- a/src/third_party/wiredtiger/src/os_win/os_open.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+++ b/src/third_party/wiredtiger/src/os_win/os_open.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;@@ -79,7 +79,7 @@ __wt_open(WT_SESSION_IMPL *session,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                dwCreationDisposition = OPEN_EXISTING;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        if (dio_type &amp;amp;&amp;amp; FLD_ISSET(conn-&amp;gt;direct_io, dio_type)) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-               f |= FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+               f |= FILE_FLAG_NO_BUFFERING;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;                direct_io = true;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;If anyone is prepared to give this a try, please let me know.&lt;/p&gt;</comment>
                            <comment id="1049918" author="nick@innsenroute.com" created="Fri, 2 Oct 2015 15:30:00 +0000"  >&lt;p&gt;I&apos;m wondering about the os_max_cache setting in WT. If that&apos;s settable then it would be interesting to try.&lt;/p&gt;</comment>
                            <comment id="1049588" author="stephen.jannin@sgcib.com" created="Fri, 2 Oct 2015 07:06:58 +0000"  >&lt;p&gt;During our activation of WiredTiger in production in July, we tried to use direct_io during few days, but we had many slow requests. Memory was not leaked anymore, but performance was catastrophic.&lt;br/&gt;
We finally went back to mmap.&lt;/p&gt;</comment>
                            <comment id="1049466" author="nick@innsenroute.com" created="Fri, 2 Oct 2015 02:53:42 +0000"  >&lt;p&gt;Retest with Direct IO off took 7h:32m vs, 5h:48m for a ~30% speed difference. Direct IO also uses 6GB less physical RAM.&lt;/p&gt;</comment>
                            <comment id="1048926" author="nick@innsenroute.com" created="Thu, 1 Oct 2015 17:44:20 +0000"  >&lt;p&gt;Ramon, there seem to be a few issues at play here:&lt;/p&gt;

&lt;p&gt;1. MongoDB memory usage &amp;gt; WT Cache size due to TCMalloc cache going over 1GB. (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20104&quot; title=&quot;WT high memory usage due to high amount of free memory accumulated by TCMalloc&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20104&quot;&gt;&lt;del&gt;SERVER-20104&lt;/del&gt;&lt;/a&gt;).&lt;br/&gt;
2. MongoDB memory usage &amp;gt; WT Cache size due to OS caching of WT files.&lt;br/&gt;
2A. On 64-bit Windows, there is no limit to how much physical memory the OS will use for file buffering, and, it appears that for certain workloads it will use all available physical memory. This cache does not appear to be released until MongoDB is shut down - even when the system is under memory stress and MongoDb is idle. This makes it impossible to constrain MongoDb using WT, and other processes on the same box may suffer memory starvation. (This ticket)&lt;br/&gt;
2B. In the case of 2A, MongoDb performance itself drops by ~25%, when compared to direct IO. Possibly this is due to a larger OS cache getting flushed continually. Perhaps this is similar to why, for certain workloads, a smaller WT cache size improves performance.&lt;/p&gt;

&lt;p&gt;To re-iterate, for a pleb such as myself, it seems odd that MongoDb soaks up &lt;b&gt;all&lt;/b&gt; the system memory, when other databases I&apos;ve used do not. Surely this cannot be the expected behavior (this ticket).&lt;/p&gt;

&lt;p&gt;If you want me to create a ticket for 2B I will, but it seems strongly correlated to this ticket.&lt;/p&gt;</comment>
                            <comment id="1048901" author="ramon.fernandez" created="Thu, 1 Oct 2015 17:24:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=nick%40innsenroute.com&quot; class=&quot;user-hover&quot; rel=&quot;nick@innsenroute.com&quot;&gt;nick@innsenroute.com&lt;/a&gt;, will you please open a separate ticket and post your results in it? Whether better memory management is needed on Windows, or whether it makes sense to make directIO the default on this platform, these are broader and different topics than the behavior/bug described in this ticket (using the &lt;tt&gt;--wiredTigerCacheSizeGB&lt;/tt&gt; to limit &lt;tt&gt;mongod&lt;/tt&gt;&apos;s memory consumption).&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1048844" author="nick@innsenroute.com" created="Thu, 1 Oct 2015 16:42:22 +0000"  >&lt;p&gt;ok - so I ran a test last night with my standard workload on 3.1.8 with WT cache size set to 4 GB and &lt;b&gt;direct IO turned on&lt;/b&gt;. Fastest run I have ever seen - finishing in 5h:48m - fantastic, consistent performance (see attached). Lots of system memory left idle.&lt;/p&gt;

&lt;p&gt;I&apos;m currently running the same test &lt;b&gt;without direct IO&lt;/b&gt;. Initially, the speeds are faster, but an hour or so into the test the speeds have dropped by 25%. The OS has paged over 6GB of WT files into memory and is bumping up against the physical memory limit.&lt;/p&gt;

&lt;p&gt;The &lt;b&gt;direct IO off&lt;/b&gt; test is still running, but when it finishes I&apos;ll upload the perf chart for that.&lt;/p&gt;</comment>
                            <comment id="1045697" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:54:35 +0000"  >&lt;p&gt;I also tested mongod releasing memory back to the OS, and I can confirm that &lt;del&gt;it does not&lt;/del&gt;. Neither Mongod nor the OS cache are released.&lt;/p&gt;

&lt;p&gt;Edit: it took a while but it did release half of the process memory, but none of the file cache memory.&lt;/p&gt;

&lt;p&gt;Killed MongoD and the WT file cache was released, and SQL Server climbed back up to 1GB (see screen shot).&lt;/p&gt;</comment>
                            <comment id="1045681" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:18:25 +0000"  >&lt;p&gt;A few notes for a baseline:&lt;/p&gt;

&lt;p&gt;I configured sql server with a 1GB cap and ran my test. See the SqlServer attachment which shows the overall memory usage is 5.2GB, with almost nothing cached by the OS and 1GB physical RAM in use by the SQL process.&lt;/p&gt;

&lt;p&gt;I ran the same test with 3.1.8 with WT cache set to 1GB (see Mongo_318 attachment). It shows the overall usage at 7GB, with 2.5 GB OS cache and MongoD using 1.3 GB.&lt;/p&gt;

&lt;p&gt;SQL = (System usage) + 1GB = 5.2 GB.&lt;br/&gt;
Mongo = (System usage) + 1.3GB + 2.5 GB = 7GB.&lt;/p&gt;

&lt;p&gt;It&apos;s difficult to illustrate but in the SQL scenario there is less memory pressure and other processes are consuming more memory. In the Mongo scenario, many of those same processes are using less memory. I suspect if that were taken into account, the system memory would be closer to 8GB for MongoD. So on this short test, MongoDb is consuming 3.8 GB of RAM with a WT cache size set to 1GB. It uses 1.8 to 2.8 GB more memory than sql server restricted to 1GB.&lt;/p&gt;

&lt;p&gt;I will re-test on my work machine which is much beefier - but from my earlier results it appears the OS cache is 5GB in that scenario. From my notes it appears on 3.1.7 with WT cache set to 3GB, the actual usage is 10 GB!&lt;/p&gt;</comment>
                            <comment id="1045025" author="nick@innsenroute.com" created="Mon, 28 Sep 2015 16:17:31 +0000"  >&lt;p&gt;And one totally unrelated comment:&lt;/p&gt;

&lt;p&gt;&quot;...Note that setting a large value for the WiredTiger cache to improve performance...&quot; isn&apos;t necessarily true. I was surprised to learn that &lt;b&gt;lowering&lt;/b&gt; the WT cache size for insert-heavy workloads had a surprisingly large impact on performance (improvement that is).&lt;/p&gt;</comment>
                            <comment id="1045019" author="nick@innsenroute.com" created="Mon, 28 Sep 2015 16:10:41 +0000"  >&lt;p&gt;A few comments:&lt;/p&gt;

&lt;p&gt;1. Use --wiredTigerCacheSizeGB to limit the wiredtiger cache. This works.&lt;br/&gt;
2. There is currently an issue with TCMalloc + MongoDb where an indeterminate amount of memory is being used in addition to the WT cache. My understanding is that TCMalloc&apos;s cache was to be limited to ~1GB, however this isn&apos;t currently the case, and significantly more may be used (there is an existing ticket for this).&lt;br/&gt;
3. The OS is caching/buffering WT&apos;s files. This is expected. What is not expected is the amount of memory the OS is using for this. There appears to be an OS cache size roughly equal to the WT cache size, although this may just be coincidence in my testing. &lt;br/&gt;
4. The OS does not appear to be releasing the WT file cache, as other processes on my test box end up getting starved for memory.&lt;br/&gt;
5. The OS does not appear to be releasing the WT file cache as physical memory is exhausted, and even with only MongoDB on the box, paging kicks in and hurts performance.&lt;br/&gt;
6. The OS cache memory is tricky to track down because it doesn&apos;t show up in task manager - it requires special tools (rammap) to show where the memory is getting soaked up.&lt;/p&gt;

&lt;p&gt;As previously mentioned, I will attempt to run some further analysis and perhaps compare this behavior with Sql Server. Is this something @Mark Benvenuto could comment on?&lt;/p&gt;</comment>
                            <comment id="1044615" author="stephen.jannin@sgcib.com" created="Mon, 28 Sep 2015 08:00:25 +0000"  >&lt;p&gt;I disagree too. We cannot use WiredTiger in these conditions.&lt;/p&gt;</comment>
                            <comment id="1044261" author="nick@innsenroute.com" created="Sat, 26 Sep 2015 20:26:22 +0000"  >&lt;p&gt;Ramon,&lt;/p&gt;

&lt;p&gt;I disagree. I&apos;ll perform further testing but I don&apos;t think this makes sense.&lt;/p&gt;</comment>
                            <comment id="1044197" author="ramon.fernandez" created="Sat, 26 Sep 2015 12:11:20 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=marc.girollet%40sgcib.com&quot; class=&quot;user-hover&quot; rel=&quot;marc.girollet@sgcib.com&quot;&gt;marc.girollet@sgcib.com&lt;/a&gt;, if my understanding of the issue description above is correct this is expected behavior: &lt;b&gt;the &lt;tt&gt;--wiredTigerCacheSizeGB&lt;/tt&gt; only limits the WiredTiger cache, not the amount of memory consumed by &lt;tt&gt;mongod&lt;/tt&gt;&lt;/b&gt;.&lt;/p&gt;

&lt;p&gt;There&apos;s currently no global limit one can set to limit the amount of memory used by &lt;tt&gt;mongod&lt;/tt&gt;. Also the OS may use additional memory for buffering, which may be released to other processes if there&apos;s memory pressure. Note that setting a large value for the WiredTiger cache to improve performance will reduce the amount of memory the OS may use for buffering, which may have a negative effect on performance.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;
</comment>
                            <comment id="1014040" author="nick@innsenroute.com" created="Tue, 25 Aug 2015 16:29:38 +0000"  >&lt;p&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;ShutDownMongoD.png&amp;#93;&lt;/span&gt; This shows the memory drop when mongod is shut down (~10G). Task manager details tab shows the working set to be ~5G.&lt;/p&gt;</comment>
                            <comment id="1013993" author="eitan.klein" created="Tue, 25 Aug 2015 15:50:50 +0000"  >&lt;p&gt;Attached file that highlight the visibility of this issue for windows users.&lt;/p&gt;</comment>
                            <comment id="1013975" author="nick@innsenroute.com" created="Tue, 25 Aug 2015 15:34:46 +0000"  >&lt;p&gt;Mods, please link this to &lt;a href=&quot;https://jira.mongodb.org/browse/WT-1990?jql=text%20~%20%22Cache%20windows%22&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/WT-1990?jql=text%20~%20%22Cache%20windows%22&lt;/a&gt; which appears to be the original ticket.&lt;/p&gt;</comment>
                            <comment id="1013907" author="nick@innsenroute.com" created="Tue, 25 Aug 2015 14:55:45 +0000"  >&lt;p&gt;I also see this behavior on my systems. Even though in task manager Mongo will appear to use 5GB (even with CacheSize set to 3GB, but that is a different issue), the actual memory usage will be close to double that. RamMap doesn&apos;t yet support windows 10 but it&apos;s easy to see that the overall system memory usage drops by close to double what is reporting by the task manager details tab when MongoD is shut down.&lt;/p&gt;

&lt;p&gt;I don&apos;t think this is the correct behavior and, for example, a capped Sql Server instance does not appear to use the OS caching in this manner (ie, a cap of 4GB = 4GB physical memory usage).&lt;/p&gt;

&lt;p&gt;I&apos;m working with Eitan on some other related issues but I&apos;ve mentioned this to him.&lt;/p&gt;</comment>
                            <comment id="999317" author="stephen.jannin@sgcib.com" created="Fri, 7 Aug 2015 13:38:18 +0000"  >&lt;p&gt;Comparing wiredTiger source Code and mmapv1 source code, I have a few remarks : &lt;/p&gt;

&lt;p&gt;1/ difference in the way we create and use memory mapped files. &lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;wiredTiger : use of  MapViewOfFile&lt;/li&gt;
	&lt;li&gt;mmapv1 : use of MapViewOfFileEx, and you specify a requested base address using &quot;getNextMemoryMappedFileLocation&quot;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;2/ wiredTiger : you never call FlushViewOfFile nor FlushFileBuffers, where I think something happens in mmapv1. Maybe in data_file_sync.cpp, where we call flushAllFiles in a loop in a thread.&lt;/p&gt;
</comment>
                            <comment id="999243" author="marc.girollet@sgcib.com" created="Fri, 7 Aug 2015 11:48:55 +0000"  >&lt;p&gt;Hi Michael,&lt;br/&gt;
Thanks for your reply , we actually are using 3.0.4 (and have tested 3.0.5) versions , both have the problem.&lt;/p&gt;

&lt;p&gt;A well stable case is leaking 1,172,768K (previous screen shot) ,updates (100 per min) queries (150 per min)&lt;br/&gt;
Net in 3MB, Net Out 16 MB , DB size on disk : 1.44 GB&lt;/p&gt;

&lt;p&gt;Attached , activity Graph , server status&lt;/p&gt;

&lt;p&gt;--------------------&lt;/p&gt;

&lt;p&gt;We managed to restrict the OS cache use (see &lt;a href=&quot;http://www.uwe-sieber.de/ntcacheset_e.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.uwe-sieber.de/ntcacheset_e.html&lt;/a&gt; ) we used SetSystemFileCacheSize.&lt;/p&gt;

&lt;p&gt;The point is that we didn&apos;t have to do this with mmap engine&lt;/p&gt;</comment>
                            <comment id="998102" author="michael.cahill" created="Thu, 6 Aug 2015 10:28:44 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=marc.girollet%40sgcib.com&quot; class=&quot;user-hover&quot; rel=&quot;marc.girollet@sgcib.com&quot;&gt;marc.girollet@sgcib.com&lt;/a&gt;, I have moved this ticket to the SERVER project, which deals with MongoDB issues.&lt;/p&gt;

&lt;p&gt;Note that the WiredTiger cache is not designed to include all sources of memory allocated by &lt;tt&gt;mongod&lt;/tt&gt;.  WiredTiger itself allocates memory for various purposes other than the page cache, such as for buffering log records and caching keys and values in cursors.&lt;/p&gt;

&lt;p&gt;To make progress with the pattern of memory use you are seeing, please include more information about the version of MongoDB you are running and the workload that is causing this behavior.  The &lt;tt&gt;direct I/O&lt;/tt&gt; configuration is included to avoid having the operating system cache filesystem buffers, but that is intended for performance tuning rather than to avoid out-of-memory issues.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="234885">SERVER-20991</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="214726">WT-1990</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="92351" name="3.1.8 direct IO off.png" size="36811" author="nick@innsenroute.com" created="Fri, 2 Oct 2015 02:50:11 +0000"/>
                            <attachment id="87478" name="HighMemoryUsage.png" size="53886" author="eitan.klein" created="Tue, 25 Aug 2015 15:50:50 +0000"/>
                            <attachment id="91741" name="MongoNoRelease_1.png" size="155276" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:59:23 +0000"/>
                            <attachment id="91742" name="MongoPartialMemRelease.png" size="161073" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 04:01:12 +0000"/>
                            <attachment id="91743" name="MongoShutdownCacheRelease.png" size="359267" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 04:06:04 +0000"/>
                            <attachment id="91740" name="OS Caching.png" size="53644" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:44:50 +0000"/>
                            <attachment id="87483" name="ShutDownMongoD.png" size="59665" author="nick@innsenroute.com" created="Tue, 25 Aug 2015 16:29:38 +0000"/>
                            <attachment id="84920" name="WT_RAMMAP.png" size="582654" author="marc.girollet@sgcib.com" created="Thu, 6 Aug 2015 09:58:29 +0000"/>
                            <attachment id="85090" name="WT_WORKLOAD.png" size="42083" author="marc.girollet@sgcib.com" created="Fri, 7 Aug 2015 11:49:07 +0000"/>
                            <attachment id="92231" name="direct_IO_ON.png" size="36331" author="nick@innsenroute.com" created="Thu, 1 Oct 2015 16:44:30 +0000"/>
                            <attachment id="91738" name="mongo_318.png" size="109512" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:06:19 +0000"/>
                            <attachment id="85089" name="serverstatus.json" size="24388" author="marc.girollet@sgcib.com" created="Fri, 7 Aug 2015 11:49:07 +0000"/>
                            <attachment id="91737" name="sqlserver.png" size="112678" author="nick@innsenroute.com" created="Tue, 29 Sep 2015 03:03:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000Z4MM2IAN]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 6 Aug 2015 10:28:44 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 17 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 17 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[Windows]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eitan.klein</customfieldvalue>
            <customfieldvalue>marc.girollet@sgcib.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>nick@innsenroute.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>stephen.jannin@sgcib.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkyc7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hscd1b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrll9r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>