<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 02:54:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-574] Detrimental performance when paging (need to reduce concurrency, use madvise and mincore)</title>
                <link>https://jira.mongodb.org/browse/SERVER-574</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;If the memory mapped pages of a file are not present in RAM then a page fault is taken to get the contents.  If there is concurrency and the requests do not have much in common then the situation gets worse as many (random) different areas will be taking page faults which causes lots of disk accesses including random seeks dropping throughput considerably.  As this slows down completion of all executing requests, it increases the chance of another request coming in and if that starts executing, it will make things even worse.  What you end up seeing is an essentially idle CPU, the I/O subsystem at 100% capacity and hard disks seeking their hearts out.&lt;/p&gt;

&lt;p&gt;The consequence is that when MongoDB hits this capacity limit, performance falls off a cliff.  There are several things that can be done to correct this:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;b&gt;Reduce&lt;/b&gt; concurrency as saturation is approached to let requests complete quicker instead of having lots of slow very long running requests&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Under POSIX the madvise system call can be used.  For example if an index or data is being sequentially read you could use madvise MADV_SEQUENTIAL|MADV_WILLNEED to suggest the kernel fill those pages in.  You can use MADV_DONTNEED on pages that won&apos;t be needed again in the near future, as that will help the kernel determine which pages can be evicted to make space for new ones.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;You can use the mincore system call to determine if a page fault will be taken for a memory range.  This is probably the best test for available concurrency (ie throttle how often you proceed when it returns false)&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment>Ubuntu 9.10, 64 bit</environment>
        <key id="11200">SERVER-574</key>
            <summary>Detrimental performance when paging (need to reduce concurrency, use madvise and mincore)</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="rogerbinns">Roger Binns</reporter>
                        <labels>
                            <label>mmapv1</label>
                    </labels>
                <created>Tue, 26 Jan 2010 21:51:03 +0000</created>
                <updated>Tue, 6 Dec 2022 05:50:56 +0000</updated>
                            <resolved>Fri, 14 Sep 2018 20:26:32 +0000</resolved>
                                    <version>1.3.1</version>
                                                    <component>MMAPv1</component>
                    <component>Performance</component>
                                        <votes>38</votes>
                                    <watches>36</watches>
                                                                                                                <comments>
                            <comment id="19009" author="rogerbinns" created="Fri, 8 Oct 2010 16:45:17 +0000"  >&lt;p&gt;@Matthias: While just watching query completion times will help, the problem with that is that throttling will affect all queries including those that could have been served out of memory immediately.&lt;/p&gt;

&lt;p&gt;For example lets say that half of queries are over the same range of data which is consequently in memory, and the other half are very random.  Throttling will affect both whereas only the random ones need to be throttled.&lt;/p&gt;

&lt;p&gt;Taking advantage of operating system calls in order to be smart about throttling is a good thing.&lt;/p&gt;</comment>
                            <comment id="18989" author="mgoetzke" created="Fri, 8 Oct 2010 06:35:55 +0000"  >&lt;p&gt;it might be possible to achieve the same warning by looking at deviations of query time over the last x seconds/minutes. this way os specific functions would not be needed (especially since they might be difficult to work into the access code).&lt;/p&gt;

&lt;p&gt;e.g.&lt;/p&gt;

&lt;p&gt;have x concurrent workers, watch speed over last x queries or seconds.&lt;br/&gt;
try with x-1 -&amp;gt; compare&lt;br/&gt;
try with x+1 -&amp;gt; compare&lt;br/&gt;
adjust x to best speed &lt;/p&gt;

&lt;p&gt;do it again after x seconds or after deviation detected&lt;/p&gt;

&lt;p&gt;limit x to be within min/max&lt;/p&gt;

&lt;p&gt;it would be a similar auto-detection mechanism as used for indices, optimizing for best worker thread size automatically&lt;/p&gt;</comment>
                            <comment id="12026" author="rogerbinns" created="Wed, 27 Jan 2010 14:51:05 +0000"  >&lt;p&gt;In a benchmark run, going from 5 concurrent worker processes to 3 &lt;b&gt;decreased&lt;/b&gt; run time from 10h1m to 7h12m.  The concurrency was killing performance!  Conversely with CouchDB 5 workers took 8h1m and 3 took 10h.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 8 Oct 2010 06:35:55 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        13 years, 19 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            13 years, 19 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>mgoetzke</customfieldvalue>
            <customfieldvalue>rogerbinns</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrpsfz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8hav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6343</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrzs33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>