<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:26:59 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31417] Improve tcmalloc when decommitting large amounts of memory</title>
                <link>https://jira.mongodb.org/browse/SERVER-31417</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;tcmalloc may occasionally release large amounts of pageheap free memory to the kernel by calling madvise. This can take seconds when the amount of memory involved is many GB. A tcmalloc internal lock is held while this happens, so this can potentially stall many threads, causing widespread latency spikes.&lt;/p&gt;

&lt;p&gt;There is no direct metric that diagnoses this (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31380&quot; title=&quot;Add metrics related to tcmalloc acquiring and decommitting memory from system&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31380&quot;&gt;&lt;del&gt;SERVER-31380&lt;/del&gt;&lt;/a&gt; would provide that), but it can be indirectly inferred to be a likely cause from the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;tcmalloc pageheap free memory decreases to near zero&lt;/li&gt;
	&lt;li&gt;tcmalloc unmapped memory increases by a corresponding amount&lt;/li&gt;
	&lt;li&gt;resident memory decreases by the same amount&lt;/li&gt;
	&lt;li&gt;system free memory increases by that amount&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="441260">SERVER-31417</key>
            <summary>Improve tcmalloc when decommitting large amounts of memory</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-perf">Backlog - Performance Team</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>RF36</label>
                            <label>former-quick-wins</label>
                            <label>perf-effort-xlarge</label>
                            <label>perf-improve-product</label>
                            <label>perf-urgency-asap</label>
                            <label>perf-value-essential</label>
                    </labels>
                <created>Thu, 5 Oct 2017 18:15:42 +0000</created>
                <updated>Tue, 30 Jan 2024 15:13:56 +0000</updated>
                                                                            <component>Internal Code</component>
                                        <votes>43</votes>
                                    <watches>93</watches>
                                                                                                                <comments>
                            <comment id="5816584" author="JIRAUSER1262765" created="Fri, 27 Oct 2023 18:18:56 +0000"  >&lt;p&gt;We are also hitting this issue in production and interested if there has been any progress. &lt;/p&gt;

&lt;p&gt;We tried increasing the release rate to 5 and 10, and found that it prevented the stalls but also  significantly impacted query latency. Setting it to 2 appeared to be a good compromise, but we haven&apos;t had a chance to test it extensively yet.&lt;/p&gt;</comment>
                            <comment id="4996674" author="JIRAUSER1271960" created="Mon, 21 Nov 2022 12:54:52 +0000"  >&lt;p&gt;We are hitting this issue (server stalls while freeing up a big amount of page_heap_free_bytes). Any progress on this issue? Do we know if increasing the TCMALLOC_RELEASE_RATE may help to mitigate this issue?&lt;/p&gt;</comment>
                            <comment id="2364121" author="henrik.ingo@10gen.com" created="Fri, 9 Aug 2019 09:07:51 +0000"  >&lt;p&gt;We recently learned that TCMALLOC_RELEASE_RATE can be used to make tcmalloc release memory more frequently to the OS. We haven&apos;t tested anything that would resemble a repro of this issue, but I could speculate that using TCMALLOC_RELEASE_RATE=10 could cause tcmalloc to call madvise more frequently and in smaller chunks.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42697&quot; title=&quot;Expose tcmalloc_release_rate via setParameter&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42697&quot;&gt;&lt;del&gt;SERVER-42697&lt;/del&gt;&lt;/a&gt; will add tcmalloc_release_rate tuning via setParameter.&lt;/p&gt;
</comment>
                            <comment id="2040737" author="bruce.lucas@10gen.com" created="Wed, 24 Oct 2018 17:12:49 +0000"  >&lt;p&gt;An related effect is that mongod&apos;s reluctance to release memory to the o/s in a timely can cause two additional problems:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;It exacerbates the effect of memory fragmentation (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33296&quot; title=&quot;Excessive memory usage due to heap fragmentation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33296&quot;&gt;SERVER-33296&lt;/a&gt;), because memory fragmentation causes an accumulation of unused memory that could be released to the o/s&lt;/li&gt;
	&lt;li&gt;If mongod allocates a large amount of memory, e.g. to process a query, do an index build, etc., that memory will not be returned to the o/s, and this can hamper recovery. See for example HELP-7990.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I mention these issues because I think a fix for this ticket is likely to help with both of those issues.&lt;/p&gt;</comment>
                            <comment id="1967212" author="ian@10gen.com" created="Sun, 5 Aug 2018 21:15:03 +0000"  >&lt;p&gt;ping &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.benvenuto&quot; class=&quot;user-hover&quot; rel=&quot;mark.benvenuto&quot;&gt;mark.benvenuto&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1840922" author="bruce.lucas@10gen.com" created="Wed, 21 Mar 2018 16:36:45 +0000"  >&lt;p&gt;This has been seen by another customer on 3.6.1 in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34027&quot; title=&quot;Production issue under high load.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34027&quot;&gt;&lt;del&gt;SERVER-34027&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1704294" author="bruce.lucas@10gen.com" created="Thu, 19 Oct 2017 21:23:52 +0000"  >&lt;p&gt;I wonder if we should surface TCMALLOC_AGGRESSIVE_DECOMMIT as a server parameter, possibly changeable at runtime, so users don&apos;t have to set an environment variable and restart to test the impact of enabling it. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mark.benvenuto&quot; class=&quot;user-hover&quot; rel=&quot;mark.benvenuto&quot;&gt;mark.benvenuto&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt;?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="514435">SERVER-34027</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="514435">SERVER-34027</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="616475">SERVER-37541</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="440307">SERVER-31380</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="497374">SERVER-33296</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25124"><![CDATA[Product Performance]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000YS3GiIAL, 500A000000YR8LRIA1, 500A000000X2Oc6IAF, 500A000000WUtuXIAT, 500A000000YTruQIAT, 500A000000YRGA2IAP, 500A000000Z4Pb0IAF, 5002K00000d9dhxQAA, 5002K00000dOsyxQAC, 5002K00000dazYfQAI, 5002K00000f156uQAA, 5002K00000hRAN9QAO, 5002K00000jdaNnQAI, 5002K00000jdhiDQAQ, 5002K00000kFBzBQAW, 5002K00000kDGRlQAO, 5002K00000msEzGQAU, 5002K00000npot1QAA, 5002K00000ocRv1QAE, 5002K00000pkJ4yQAE, 5002K00000po50cQAA, 5002K00000s0JejQAE, 5002K00000scZXcQAM, 5002K00000wVWoTQAW, 5002K00000wZMS0QAO, 5002K00000zdObFQAU, 5002K000010F1cjQAC, 5002K000011eYtcQAE, 5006R00001l8d7FQAQ, 5006R00001l86X5QAI, 5006R00001lB0s2QAC, 5006R00001lRtI2QAK, 5006R00001mtbjCQAQ, 5006R00001m6xoSQAQ, 5006R00001m82FyQAI, 5006R00001nq5oyQAA, 5006R00001nqc4tQAA, 5006R00001oIkGdQAK, 5006R00001ox5uKQAQ, 5006R00001oxugJQAQ, 5006R00001pmhLFQAY, 5006R00001qE2uHQAS, 5006R00001qE7NrQAK, 5006R00001pnMxUQAU, 5006R00001qo8u9QAA, 5006R00001r7aRSQAY, 5006R00001rZvfCQAS, 5006R00001rcO8dQAE, 5006R00001rc7jxQAA, 5006R00001s9FC6QAM, 5006R00001s9NI3QAM, 5006R00001s7SjIQAU, 5006R00001sDJpKQAW, 5006R00001sE9OlQAK, 5006R00001ss5j9QAA, 5006R00001tJr7YQAS, 5006R00001sttwGQAQ, 5006R00001tL26eQAC, 5006R00001ts0F3QAI, 5006R00001oyI1BQAU, 5006R00001tvUJEQA2, 5006R00001v7DDEQA2, 5006R00001xKTfEQAW, 5006R00001yjjOwQAI, 5006R00001ykjpmQAA]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 5 Oct 2017 20:17:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        14 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/PERF-1846'>PERF-1846</a></s>, <s><a href='https://jira.mongodb.org/browse/PM-2029'>PM-2029</a></s>, <a href='https://jira.mongodb.org/browse/PM-1812'>PM-1812</a>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            14 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-perf</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>henrik.ingo@mongodb.com</customfieldvalue>
            <customfieldvalue>ian.springer@salesforce.com</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>jose@netlify.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hth5j3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hyaawv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2910">Dev Tools 2019-05-06</customfieldvalue>
    <customfieldvalue id="2942">Dev Tools 2019-04-22</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htgrmv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>