<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:05:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4301] Why does Mongo become slower when RAM is about used up</title>
                <link>https://jira.mongodb.org/browse/SERVER-4301</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When I continuously insert/select/update a collection, one thread for each operation, the whole performance (opcount) drops when one of the shards&apos; RAM is about used up. &lt;br/&gt;
The _id is increased from 0 for insert, and select/update operations are based on random  _id. I know that the performance is doomed to drop. But I want to know the reasons. &lt;br/&gt;
1. Since MongoDB flushes data from memory to disk once a minute, although RAM is about used up, only some dirty data in the memory need to be flushed, right? We call the dirty data &quot;new data&quot;. Since OS&apos;s memory management is used, LRU algorithm will free up memory space for new data, because the majority of the used memory is not fresh data and can be simply destroyed without losing any information. So, no more data swap is caused by insert when RAM is used up and I guess insert performance should NOT drop, but why did it??? Is it caused by cost of memory allocation (internal operation of insert) or just disk flush or something else???&lt;br/&gt;
2. I can understand that more disk read may be needed when the memory is near limitation, because the work set is larger than the memory size, so some of data will need be loaded from disk from time to time. When the ratio of working set size over the RAM size increases, the rate of disk read need grow as well, since more frequent data swap is needed. Is that right??? &lt;br/&gt;
3. On the other hand, the non-optimized global lock of MongoDB will block read and other write operations when one write is slow or even blocked. So, I think when RAM is almost used up, more global lock caused by slow insert will further cause the whole read&amp;amp;write performance drop, right??? BTW, will read lock block write operations? &lt;/p&gt;

&lt;p&gt;I also attache a test file here, and it is so appreciated that you can give me some help. You can quick jump to the &quot;Observe the cluster&quot; part to get direct information of my questions.&lt;/p&gt;</description>
                <environment>4 shards on Windows Server 2003, not replica set but single node is used as shard, C++ driver: aposto/mongodb-cxx-windows-driver (&lt;a href=&quot;https://github.com/aposto/mongodb-cxx-windows-driver/contributors&quot;&gt;https://github.com/aposto/mongodb-cxx-windows-driver/contributors&lt;/a&gt;) compiled on VS2005.</environment>
        <key id="25032">SERVER-4301</key>
            <summary>Why does Mongo become slower when RAM is about used up</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="tad">Tad Marshall</assignee>
                                    <reporter username="wlu@microstrategy.com">wei lu</reporter>
                        <labels>
                            <label>Windows</label>
                            <label>insert</label>
                    </labels>
                <created>Thu, 17 Nov 2011 03:00:08 +0000</created>
                <updated>Wed, 15 Aug 2012 14:04:18 +0000</updated>
                            <resolved>Sun, 4 Mar 2012 21:45:06 +0000</resolved>
                                    <version>2.0.0</version>
                                                    <component>Internal Client</component>
                    <component>Performance</component>
                    <component>Testing Infrastructure</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="95113" author="tad" created="Sun, 4 Mar 2012 21:44:45 +0000"  >&lt;p&gt;Consolidating closely related tickets into one.&lt;/p&gt;</comment>
                            <comment id="75511" author="tad" created="Thu, 22 Dec 2011 18:34:22 +0000"  >&lt;p&gt;I tested running xperf as described above and at first glance I can&apos;t see that it reveals anything.  You will get more useful results if you limit the time interval as much as possible and don&apos;t run anything else on the system while xperf is collecting data.  In 45 minutes of logging, I created a 4.5 GB trace file but because I was using my system while the test was running, it shows all my activity, not just mongod.exe.  I&apos;m not sure if xperf will be helpful in diagnosing the memory issues.&lt;/p&gt;</comment>
                            <comment id="70794" author="wlu@microstrategy.com" created="Mon, 5 Dec 2011 03:07:45 +0000"  >&lt;p&gt;I am just back from a short vacation, and sorry for the late responding. &lt;br/&gt;
It is right that I am using Windows Server 2003 as server. Are there special tricks I need to take care as for Server 2003, because it is specially pointed out in the question?&lt;/p&gt;</comment>
                            <comment id="70050" author="tad" created="Thu, 1 Dec 2011 14:24:02 +0000"  >&lt;p&gt;In starting to talk to Microsoft about these issues (memory usage, degraded performance) they had some suggestions for research:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Can you please confirm what version of server your customers are using?  Is this on Windows Server 2003? &lt;/p&gt;

&lt;p&gt;Below is the info on how to get tracing info &#8211; this should help investigate the issue&lt;/p&gt;

&lt;p&gt;1)  Install Xperf on the server with the issue. (it comes with the Windows Performance Toolkit from the Windows SDK)&lt;/p&gt;

&lt;p&gt;2)  On a command prompt, launch the following command&lt;br/&gt;
  a.  Xperf.exe &#8211;on base+latency+fileio &#8211;stackwalk profile+filewrite+fileread+fileflush&lt;/p&gt;

&lt;p&gt;3)  Reproduce the issue you&#8217;re facing&lt;/p&gt;

&lt;p&gt;4)  Run the following command:&lt;br/&gt;
  a.  Xperf.exe &#8211;d mytrace.etl&lt;/p&gt;

&lt;p&gt;5)  View the trace: xperf.exe mytrace.etl&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I haven&apos;t tried this yet, but I will and will report what I learn.  You could try it on your machines and see if it tells you something useful, thanks!&lt;/p&gt;</comment>
                            <comment id="69712" author="wlu@microstrategy.com" created="Wed, 30 Nov 2011 05:16:15 +0000"  >&lt;p&gt;The &quot;used up&quot; means that res memory size is about to reach the physical RAM size, and page swap does indeed occurs. The thing is, then more pages swapped in this scenario, the whole performance dropped, as I mentioned in the document. &lt;br/&gt;
I did a test by calling EmptyWorkingSet function when the res memory reaches 6G. A chart illustrating performance is attached here.&lt;/p&gt;</comment>
                            <comment id="69707" author="tad" created="Wed, 30 Nov 2011 04:37:25 +0000"  >&lt;p&gt;Thanks for your note.  You may be deeper into the relevant code than I am at this point.  But it makes little sense that a call to SetProcessWorkingSetSize() should be able to free memory that we have &quot;locked&quot; somehow, and if we haven&apos;t &quot;locked&quot; it (I&apos;m using the word &quot;locked&quot; casually, there may not be any explicit lock) then I don&apos;t understand why it isn&apos;t just &quot;taken&quot; by Windows when it needs physical RAM.  Memory should not be &quot;used up&quot; when it can simply be paged out to a file.&lt;/p&gt;

&lt;p&gt;You may be right that we need to explicitly control the size of the views that we create.  A theory we have had is that the OS knows how to manage memory and we should let it do its job, but we may need to revisit that logic.  I&apos;d like to dig a little deeper (maybe catch up with you!) before we start to change code.  If you have specific bits of code that you&apos;d like to direct me to, that would be great.&lt;/p&gt;

&lt;p&gt;I&apos;m halfway into debugging a different problem with memory mapped files on Windows and will be distracted by that for a day or two more, but then I&apos;d like to get to the bottom of the out-of-memory issues that you and others have seen on Windows.  Both performance and stability are not what they should be under some workloads, and we can and should fix it.&lt;/p&gt;</comment>
                            <comment id="69689" author="wlu@microstrategy.com" created="Wed, 30 Nov 2011 02:03:26 +0000"  >&lt;p&gt;Thank you for your responding. Actually, I&#12288;&#65362;&#65349;&#65345;&#65348;&#12288;&#65364;&#65352;&#65349;&#12288;&#65363;&#65359;&#65365;&#65362;&#65347;&#65349;&#12288;&#65347;&#65359;&#65348;&#65349;&#65363;&#12288;&#65359;&#65350;&#12288;&quot;db_10.sln&quot; recently&#65292;&#12288;&#65349;&#65363;&#65360;&#65349;&#65347;&#65353;&#65345;&#65356;&#65356;&#65369;&#12288;&#65347;&#65359;&#65348;&#65349;&#65363;&#12288;&#65362;&#65349;&#65356;&#65345;&#65366;&#65349;&#65358;&#65364;&#12288;&#65364;&#65359;&#12288;&#65325;&#65349;&#65357;&#65359;&#65362;&#65369;&#12288;&#65357;&#65345;&#65360;&#65360;&#65349;&#65348;&#12288;&#65350;&#65353;&#65356;&#65349;&#65363;&#65294;&#12288;&#65321;&#12288;&#65350;&#65353;&#65358;&#65348;&#12288;&#65364;&#65352;&#65345;&#65364;&#12288;&#65366;&#65353;&#65349;&#65367;&#65363;&#12288;&#65359;&#65350;&#12288;&#65345;&#12288;&#65350;&#65353;&#65356;&#65349;&#12288;&#65353;&#65363;&#12288;&#65357;&#65345;&#65360;&#65360;&#65349;&#65348; into memory address &#65367;&#65352;&#65349;&#65358;&#12288;&#65367;&#65349;&#12288;&#65353;&#65358;&#65363;&#65349;&#65362;&#65364; or select a document. However, the memory is not unmapped until we shutdown the process. So the memory is used up as more and more documents are affected. I think, if memory pages which are not affected for a long time are unmapped, the memory usage may be better managed. But I don&apos;t know whether it is possible to do that...&lt;br/&gt;
What I currently do is to just empty the working set of &quot;mongod&quot; process when the res memory is up to a threshold, (EmptyWorkingSet function is called by another process). &lt;/p&gt;</comment>
                            <comment id="69527" author="tad" created="Tue, 29 Nov 2011 14:57:57 +0000"  >&lt;p&gt;Sorry for the delay in responding.  I have an ongoing project to get to the bottom of multiple issues with memory usage on Windows.  Unless I can determine something that we are not doing right, we may need to follow up with Microsoft to determine what is wrong.&lt;/p&gt;

&lt;p&gt;The short story is that our use of memory mapped files should not present Windows with the kind of memory demand that we see.&lt;/p&gt;

&lt;p&gt;When a normal process uses memory, it just allocates it from a pool using an API that eventually turns into a call to HeapAlloc(), or it reserves private memory for itself using VirtualAlloc() and then &quot;commits&quot; it by writing into it.  In both of these cases, to reclaim the physical memory, Windows must page out that data to the page file.&lt;/p&gt;

&lt;p&gt;This is not the case for MongoDB&apos;s memory mapped files.  When Windows needs physical RAM for any process, it can page our memory mapped file out to the file itself and free the memory that way.  Memory mapped files do not consume page file space: they act as their own page files.&lt;/p&gt;

&lt;p&gt;This doesn&apos;t work the way it should under load.  Somehow, Windows ends up consuming more memory, leaving less for other processes, and even leaving less for MongoDB itself.  Under extreme load, &quot;free&quot; memory dwindles until everything becomes dog slow and things start to fail.&lt;/p&gt;

&lt;p&gt;This isn&apos;t a very good answer and I apologize for that, but my research so far hasn&apos;t found the precise place where a fix can be made, by either us or Microsoft.  I&apos;ll update the bug with more and better information as soon as I can, thank you for your patience!&lt;/p&gt;</comment>
                            <comment id="67355" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 12:00:58 +0000"  >&lt;p&gt;cp.zip contains executable file, but I am not sure whether it works on your machine. You can run the python script;&lt;br/&gt;
MongoDBThread.cpp is the source code file describes how APIs are called: Ln93 select, Ln 230 insert, Ln 305 update.&lt;br/&gt;
MongoTestData.dat contains the content of the document I inserted in the test. All documents are the same. In UPDATE, I add another field to the document.&lt;/p&gt;</comment>
                            <comment id="67353" author="tad" created="Thu, 17 Nov 2011 11:38:17 +0000"  >&lt;p&gt;If this is using dummy (non-proprietary) data, would it be possible for you to upload your code so that we could reproduce your results?  Performance might be affected by document sizes and index paging, and we could give better advice if we could see the details of what is driving performance in your specific test cases.  Thanks!&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="32153">SERVER-5194</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="13856" name="MongoDB Cluster Performance.docx" size="78325" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 03:00:09 +0000"/>
                            <attachment id="13857" name="MongoDB Cluster Performance.pdf" size="346489" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 03:00:09 +0000"/>
                            <attachment id="13860" name="MongoDBThread.cpp" size="10895" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 12:00:58 +0000"/>
                            <attachment id="13861" name="MongoTestData.dat" size="1992" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 12:00:58 +0000"/>
                            <attachment id="13859" name="cp.zip" size="796043" author="wlu@microstrategy.com" created="Thu, 17 Nov 2011 12:00:58 +0000"/>
                            <attachment id="13968" name="empty.JPG" size="28764" author="wlu@microstrategy.com" created="Wed, 30 Nov 2011 05:16:15 +0000"/>
                            <attachment id="13967" name="empty.JPG" size="28764" author="wlu@microstrategy.com" created="Wed, 30 Nov 2011 05:14:35 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 17 Nov 2011 11:38:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 50 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 50 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>tad</customfieldvalue>
            <customfieldvalue>wlu@microstrategy.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrolfj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrit33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23365</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkspb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>