<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:04:24 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3911] memory leak with journaling in windows</title>
                <link>https://jira.mongodb.org/browse/SERVER-3911</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I created a simple test to just insert into a 1mb capped collection&lt;br/&gt;
windows &lt;br/&gt;
 with journaling&lt;br/&gt;
  memory growth after inserting 2m items is around 4gb and if I let it continue it will use all available memory&lt;/p&gt;

&lt;p&gt; without journaling &lt;br/&gt;
  memory quickly stops growing no mater how long test runs.&lt;/p&gt;

&lt;p&gt;on linux &lt;br/&gt;
 with journaling &lt;br/&gt;
   memory quickly stops growing no matter how long test is run.&lt;br/&gt;
 without journaling&lt;br/&gt;
   memory quickly stops growing no matter how long test is run.&lt;/p&gt;


&lt;p&gt;I tried in 1.8.3 version with same problem.&lt;/p&gt;</description>
                <environment>windows 64bit </environment>
        <key id="22599">SERVER-3911</key>
            <summary>memory leak with journaling in windows</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.mongodb.org/images/icons/priorities/blocker.svg">Blocker - P1</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="tad">Tad Marshall</assignee>
                                    <reporter username="pbrumm">Pete Brumm</reporter>
                        <labels>
                            <label>Windows</label>
                    </labels>
                <created>Tue, 20 Sep 2011 18:39:12 +0000</created>
                <updated>Wed, 15 Aug 2012 14:02:57 +0000</updated>
                            <resolved>Sun, 4 Mar 2012 21:56:19 +0000</resolved>
                                    <version>2.0.0</version>
                                                    <component>Storage</component>
                                        <votes>1</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="95114" author="tad" created="Sun, 4 Mar 2012 21:55:57 +0000"  >&lt;p&gt;I did not find evidence of a memory leak after running this test twice for days each time, but Windows&apos;s poor working set management explains all the problems that I did see.  In an overcommit situation, Windows will give mongod.exe&apos;s memory mapped files enough memory to cripple the system.&lt;/p&gt;</comment>
                            <comment id="88296" author="tad" created="Tue, 14 Feb 2012 18:14:27 +0000"  >&lt;p&gt;I left the Ruby program running for three days, inserting into the capped collection on a primary with two secondaries, all with limited-size oplogs (333 MB each).  Memory consumption after three days was the same as it was after a few hours.  I do not believe that there is an ongoing cumulative memory leak.  There may be excessive memory consumption and the behavior in overcommit situations may indeed cause all available memory to be exhausted, but it isn&apos;t a &quot;leak&quot; in the sense that I understand that word.&lt;/p&gt;</comment>
                            <comment id="87450" author="pbrumm" created="Fri, 10 Feb 2012 22:24:06 +0000"  >&lt;p&gt;Great response.   Thanks for the detail.&lt;/p&gt;

&lt;p&gt;I wrote the ruby script the way I did because my expectation is that oplog size + mongodb instance memory would be the max size that I would expect memory to grow.    After that it would stabilize.   I just never saw that stabilize and either of your potential issues would show the same behavior. &lt;/p&gt;

&lt;p&gt;and a memory leak or oplog/journaling leak would show the same as that is where the only data is being created in this example.&lt;/p&gt;

&lt;p&gt;My guess is the second that it is a memory map release issue because it went away without journaling.&lt;/p&gt;
</comment>
                            <comment id="87440" author="tad" created="Fri, 10 Feb 2012 21:52:13 +0000"  >&lt;p&gt;In case I&apos;ve confused things, I&apos;m in complete agreement with you that Windows behaves quite differently when journaling is used compared to no journaling, and also that Windows and Linux behave quite differently when journaling is used.  I&apos;d like to improve what I can, but some of this behavior is built into the products by design.  When journaling is used, we memory map the database twice and virtual memory usage is doubled on both Windows and Linux.  In Linux, one copy of the database is mapped copy-on-write, but we can&apos;t do this on Windows because Windows wants to reserve page file space for copy-on-write memory mapped files: a 500 GB database would require 500 GB of page file if we tried this.  So, on Windows, we set pages to copy-on-write individually instead of all at once.  Once they are flushed to disk, they are set back to read/write.  In theory, this should return the memory to its initial state: memory mapped twice but only in memory once.  Where this isn&apos;t working right, I need to fix it.&lt;/p&gt;

&lt;p&gt;An actual memory leak would behave differently.  Even with limited data, it would consume memory forever.  What I&apos;m saying is that I don&apos;t think this is happening (or at least I haven&apos;t been able to make it happen yet).&lt;/p&gt;

&lt;p&gt;That still leaves the first problem, where copy-on-write memory isn&apos;t converted back into read/write memory the way it is supposed to be.  I&apos;m trying to figure out if I have one problem or two here.  I&apos;ll let my test run over the weekend and see what it looks like after that.  Right now, it&apos;s at 7 million insertions and still at 3.09 GB RAM, the same amount it was using at 3.75 million.&lt;/p&gt;

&lt;p&gt;Thanks for your patience.&lt;/p&gt;</comment>
                            <comment id="87412" author="pbrumm" created="Fri, 10 Feb 2012 20:48:35 +0000"  >&lt;p&gt;When I ran it I am not sure what the oplog was set too or whether it was larger than ram.&lt;/p&gt;

&lt;p&gt;Free memory was consumed very in with a very steady growth.  If I disabled journaling on the two set replica the issue went away.&lt;/p&gt;

&lt;p&gt;When I ran the same script against a similar setup on linux with or without journaling the memory usage didn&apos;t show this pattern.&lt;/p&gt;

&lt;p&gt;We ran it on a test box with 48gb of ram and a 16gb opslog.   and it definitely grew past the 16gb (where I expected it to stop) and eventually filled up the 48gb.&lt;/p&gt;

&lt;p&gt;Once your test completes, rerun without journaling and see if the memory usage changes.&lt;/p&gt;</comment>
                            <comment id="87399" author="tad" created="Fri, 10 Feb 2012 20:23:45 +0000"  >&lt;p&gt;Is it possible that what is seen here is not a memory leak but just a very high (and stable, if enough RAM is available) usage of memory?  In putting together a test with one primary and two secondaries (replica set) in a 4 GB VirtualBox instance of Windows Server 2008 R2, I had to limit the oplog sizes to get performance that was better than terrible.  Once I did that, memory usage is very high (800+ MB in each mongod) but stable.  Without limiting the oplog size it might also have tried to stabilize, but at a level well beyond physical RAM, leading to out-of-memory failures before anything like &quot;stable&quot; could have been achieved.&lt;/p&gt;

&lt;p&gt;Running Pete&apos;s Ruby test with oplog sizes set to 333 MB in each instance, memory usage rose steadily to just over 3 GB (in a 4 GB VirtualBox instance) and has not budged in the last hour.  It&apos;s not acting like a leak.&lt;/p&gt;

&lt;p&gt;I&apos;ll let it run for a few days (if it doesn&apos;t hang again) and see what it looks like then.  It&apos;s at 3.75 million insertions right now.&lt;/p&gt;</comment>
                            <comment id="87107" author="tad" created="Thu, 9 Feb 2012 22:57:35 +0000"  >&lt;p&gt;Overnight test hung (Ruby program stuck after 231 million insertions) and I need to install code to debug it ... more info once I have a full debug environment set up.&lt;/p&gt;</comment>
                            <comment id="86700" author="tad" created="Wed, 8 Feb 2012 21:59:42 +0000"  >&lt;p&gt;Starting a new test using Pete&apos;s Ruby program (writes to capped collection) on Windows Server 2008 R2 with a primary and two secondaries all running in the same instance, and I&apos;ll leave it running overnight.  Testing this before, without the replica set, I was unable to duplicate the memory leak.&lt;/p&gt;</comment>
                            <comment id="86336" author="rozacki@gmail.com" created="Tue, 7 Feb 2012 23:33:54 +0000"  >&lt;p&gt;Is anybody working on it now ?&lt;/p&gt;</comment>
                            <comment id="78626" author="tad" created="Mon, 9 Jan 2012 20:33:38 +0000"  >&lt;p&gt;Sorry about the delay.  I will retest with replica sets and see if I can get this to happen.&lt;/p&gt;</comment>
                            <comment id="75649" author="pbrumm" created="Fri, 23 Dec 2011 14:19:44 +0000"  >&lt;p&gt;I don&apos;t have access to the environment to provide the output.  &lt;/p&gt;

&lt;p&gt;I was also watching the performance monitor for free memory on the box.   that was the main thing that was shrinking.  Eventually it would run out, but only if journaling was enabled.&lt;/p&gt;

&lt;p&gt;try adding a replica set if you have the time.&lt;/p&gt;

&lt;p&gt;I had an 8gb box with 4gb free and it would quickly consume that 4gb.&lt;/p&gt;

&lt;p&gt;I had a win7 box with a 2 instance replica set.   we also reproduced on a production 48gb 3 server replica set running win 2008 server.&lt;/p&gt;

&lt;p&gt;There was a difference running with journaling enabled or disabled.&lt;/p&gt;



</comment>
                            <comment id="75634" author="tad" created="Fri, 23 Dec 2011 11:01:58 +0000"  >&lt;p&gt;@Pete, yes, I have journaling enabled (by default in this version) but no replica sets.  Can you post the output from db.serverStatus() and db.stats() to help me duplicate the bug?  Thanks!&lt;/p&gt;</comment>
                            <comment id="75621" author="kobevaliant" created="Fri, 23 Dec 2011 08:00:32 +0000"  >&lt;p&gt;I believe I encountered the same bug. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4522&quot; title=&quot;inserts take long time after several hours insert operation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4522&quot;&gt;&lt;del&gt;SERVER-4522&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="75523" author="pbrumm" created="Thu, 22 Dec 2011 19:02:23 +0000"  >&lt;p&gt;are you running with journaling enabled?&lt;br/&gt;
That was the only time it would grow.&lt;/p&gt;

&lt;p&gt;It is also possible that both the environments had a 2 or 3 server replica set. &lt;/p&gt;</comment>
                            <comment id="75519" author="tad" created="Thu, 22 Dec 2011 18:50:31 +0000"  >&lt;p&gt;I retested using 2.0.0 (still on Windows 7 64-bit) and let it run longer.  I&apos;m still not seeing the memory growth you describe.  I let it go to 42,000,000 iterations and memory usage was flat after the first 1,000,000.&lt;/p&gt;</comment>
                            <comment id="75252" author="pbrumm" created="Wed, 21 Dec 2011 22:27:38 +0000"  >&lt;p&gt;It was a pretty quick growth.  resource monitor showed a steady growth from the beginning and never stopped.     I usually stopped at a count of 2M&lt;/p&gt;

&lt;p&gt;your env is fine.   try with the older 2.0.0.   Maybe it was fixed by another commit.   &lt;/p&gt;

</comment>
                            <comment id="75242" author="tad" created="Wed, 21 Dec 2011 22:18:43 +0000"  >&lt;p&gt;@Pete, I&apos;m running your Ruby program on Windows 7 64-bit against our current development version and I&apos;m not seeing the continually growing memory usage you see.  How long do you need to let it run to see problems?  The displayed counter is now at 5000000 and memory usage has not increased at all from what it was at 1000000.  Process Explorer, Task Manager, Resource Monitor and VMMap are all showing flat (horizontal) graphs for all memory components for the mongod.exe process.  I can try again on Server 2008 R2 and I can try your version (2.0.0) instead of the 2.1.0-pre I&apos;m testing, but I had expected to be able to reproduce it on Windows 7.&lt;/p&gt;</comment>
                            <comment id="61248" author="pbrumm" created="Tue, 18 Oct 2011 20:10:00 +0000"  >&lt;p&gt;try running the attached script.   &lt;/p&gt;

&lt;p&gt;in it the database doesn&apos;t grow.&lt;/p&gt;

&lt;p&gt;it is inserting on a capped collection.     &lt;/p&gt;


&lt;p&gt;the resident memory is growing quickly&lt;/p&gt;</comment>
                            <comment id="61235" author="dwight_10gen" created="Tue, 18 Oct 2011 19:45:50 +0000"  >&lt;p&gt;virtual memory should equal the size of the datafiles (or bigger) - and that is benign.&lt;/p&gt;

&lt;p&gt;what stastic is growing?  what is Private Bytes value?  can you post a perfmon output?&lt;/p&gt;

&lt;p&gt;are you getting a specific error message in the mongo log?&lt;/p&gt;</comment>
                            <comment id="60920" author="redbeard0531" created="Mon, 17 Oct 2011 19:27:43 +0000"  >&lt;p&gt;@Dwight: Could you take a look at this? Not sure if this is just an artifact of how mem usage is displayed on windows or if it is due to the way the private view is handled differently on windows&lt;/p&gt;</comment>
                            <comment id="57297" author="pbrumm" created="Wed, 28 Sep 2011 15:47:57 +0000"  >&lt;p&gt;any update on this?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="32153">SERVER-5194</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="13199" name="mongo_memory.rb" size="472" author="pbrumm" created="Tue, 20 Sep 2011 18:39:12 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 17 Oct 2011 19:27:43 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 50 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 50 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[Windows]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>rozacki@gmail.com</customfieldvalue>
            <customfieldvalue>dwight@mongodb.com</customfieldvalue>
            <customfieldvalue>kobevaliant</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
            <customfieldvalue>pbrumm</customfieldvalue>
            <customfieldvalue>tad</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hroq13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg3j3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7474</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0abj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>