<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:26:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31177] MongoDB consumes all free memory, leading to throttled replication</title>
                <link>https://jira.mongodb.org/browse/SERVER-31177</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a small three member replicaset running MongoDB 3.4.1:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Primary. Physical local server, Windows Server 2012, 64 GB RAM, 6 cores. Hosted in Scandinavia.&lt;/li&gt;
	&lt;li&gt;Secondary. Amazon EC2, Windows Server 2016, r4.2xlarge, 61 GB RAM, 8 vCPUs. Hosted in Germany.&lt;/li&gt;
	&lt;li&gt;Arbiter. Tiny cloud based Linux instance.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The WiredTiger cache has its default size of around 50% of RAM, so the mongod process consumes around 32 GB in our case. Additionally, over time, MongoDB uses up all free memory via the filesystem cache (memory mapping). This is expected behavior AFAIK.&lt;/p&gt;

&lt;p&gt;However, what we are seeing is that once the amount of available memory on the server drops below 1-4 % (by Windows&apos; definition of &quot;Available memory&quot;), the replication speed from the primary to the secondary instance is being throttled / capped at just over 20 MBit/s. I.e., replication never goes above that speed, and if there is more data to replicate, it will queue up and result in replication lag.&lt;/p&gt;

&lt;p&gt;This is not a pure bandwidth issue; for example while throttling is taking place, we can transfer data between the two servers over FTP at far more than 20 MBit/s.&lt;/p&gt;

&lt;p&gt;To prove that low memory is causing this throttling, we ran a small script that allocated and freed around 10 GB memory on the server. Since there was almost no available memory, this memory was reallocated mainly from the filesystem cache, and in part form the mongod process. Immediately the throttling stopped, as shown in the attached screenshot, and replication occurred at full speed. This lasted for around 3 days until MongoDB had consumed all free memory through the filesystem cache, at which point replication was again throttled.&lt;/p&gt;

&lt;p&gt;This is 100% reproducible, and in fact the workaround we are currently resorting to, to avoid replication lag.&lt;/p&gt;

&lt;p&gt;We haven&apos;t found a way to configure the amount of memory used for memory mapping, and are currently thinking that this must be a bug within MongoDB? We haven&apos;t found anything useful in the logs explaining why throttling takes place, and tried looking for duplicates for this bug without success.&lt;/p&gt;</description>
                <environment></environment>
        <key id="429777">SERVER-31177</key>
            <summary>MongoDB consumes all free memory, leading to throttled replication</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="mark.agarunov">Mark Agarunov</assignee>
                                    <reporter username="mbl54">Max Bennedich</reporter>
                        <labels>
                    </labels>
                <created>Wed, 20 Sep 2017 13:59:21 +0000</created>
                <updated>Tue, 17 Oct 2017 20:10:59 +0000</updated>
                            <resolved>Mon, 25 Sep 2017 15:31:06 +0000</resolved>
                                    <version>3.4.1</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1680530" author="mbl54" created="Mon, 25 Sep 2017 07:30:29 +0000"  >&lt;p&gt;The upgrade to 3.4.9 indeed solved the problem! The server memory consumption stays at a constant ~55%. Thanks for looking into this so quickly.&lt;/p&gt;</comment>
                            <comment id="1678848" author="mark.agarunov" created="Thu, 21 Sep 2017 16:31:50 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mbl54&quot; class=&quot;user-hover&quot; rel=&quot;mbl54&quot;&gt;mbl54&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the information. I&apos;ll leave this ticket open until you can confirm that this has resolved the issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1678670" author="mbl54" created="Thu, 21 Sep 2017 14:06:10 +0000"  >&lt;p&gt;Thanks for the update! I have upgraded our members to version 3.4.9. In the past, it has takes a few days for all memory to be consumed, so I will need until next week probably before I can tell you whether this solved the problem.&lt;/p&gt;</comment>
                            <comment id="1678139" author="mark.agarunov" created="Wed, 20 Sep 2017 21:05:40 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mbl54&quot; class=&quot;user-hover&quot; rel=&quot;mbl54&quot;&gt;mbl54&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing this information. Looking over this I suspect this may be due to &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2670&quot; title=&quot;Inefficient I/O when read full DB (poor readahead)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2670&quot;&gt;&lt;del&gt;WT-2670&lt;/del&gt;&lt;/a&gt;, which can cause the system to not automatically unmap memory once it has been used. Luckily this has been fixed in MongoDB 3.4.2, so my recommendation would be to upgrade to the latest version of MongoDB if possible and see if this resolves the issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1677975" author="mbl54" created="Wed, 20 Sep 2017 18:38:06 +0000"  >&lt;p&gt;Thanks for looking into this! I am attaching logs and additional screenshots from an event of interest at 2017-09-18 22:40 CEST.&lt;/p&gt;

&lt;p&gt;By 2017-09-18, we had experienced replication throttling for around 3 weeks nonstop (which is when we last restarted MongoDB).&lt;/p&gt;

&lt;p&gt;At 2017-09-18 22:26 we started a Java program which slowly allocated 20 GB memory on the primary instance. This memory was all released at 22:38. The throttling stopped immediately (you can see in the graph that we had speeds well above 20 MBit/s already at 22:42).&lt;/p&gt;

&lt;p&gt;Note: You may see some fluctuations in network traffic in the graph, and change in traffic pattern in the logs, between 2017-09-18 13:10 and 2017-09-18 15:01. During this period we switched the primary over to the AWS instance.&lt;/p&gt;

&lt;p&gt;Also a note about time zones:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Attached graphs and timestamps in this message are given in CEST time zone.&lt;/li&gt;
	&lt;li&gt;All MongoDB log time stamps are one hour earlier, i.e. the event described above at 22:40 should be looked for around &quot;21:40:00.000+0100&quot;.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1677901" author="mark.agarunov" created="Wed, 20 Sep 2017 17:43:13 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mbl54&quot; class=&quot;user-hover&quot; rel=&quot;mbl54&quot;&gt;mbl54&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve generated a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/8f0b853b-1164-41bf-981c-12f1f08e8e7f.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; so that you can send us this data privately. Files uploaded to the portal can only be accessed by MongoDB.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1677809" author="mark.agarunov" created="Wed, 20 Sep 2017 16:36:31 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mbl54&quot; class=&quot;user-hover&quot; rel=&quot;mbl54&quot;&gt;mbl54&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report. To get a better idea of why the memory usage is limiting replication speed, could you please provide the following:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;The complete logs from all &lt;tt&gt;mongod&lt;/tt&gt; instances when this issue is present,&lt;/li&gt;
	&lt;li&gt;An archive (tar or zip) of the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory from all &lt;tt&gt;mongod&lt;/tt&gt; instances.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;This should give us some insight into what may be causing this.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="289834">WT-2670</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="165972" name="mongodb-throttling-3.PNG" size="34888" author="mbl54" created="Wed, 20 Sep 2017 14:12:46 +0000"/>
                            <attachment id="166006" name="mongodb-throttling-5.PNG" size="45166" author="mbl54" created="Wed, 20 Sep 2017 18:35:53 +0000"/>
                            <attachment id="166005" name="mongodb-throttling-6.PNG" size="25294" author="mbl54" created="Wed, 20 Sep 2017 18:35:53 +0000"/>
                            <attachment id="166007" name="mongodb-throttling-7.PNG" size="41957" author="mbl54" created="Wed, 20 Sep 2017 18:35:53 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 20 Sep 2017 16:36:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 20 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 20 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>mark.agarunov</customfieldvalue>
            <customfieldvalue>mbl54</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htf6wn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht70qf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hteszz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>