<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:53:38 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20248] Memory growth in __wt_session_get_btree in __checkpoint_worker under WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-20248</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This is ticket is related to the latest issue discussed on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17456&quot; title=&quot;Mongodb 3.0 wiredTiger storage engine memory usage too high.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17456&quot;&gt;&lt;del&gt;SERVER-17456&lt;/del&gt;&lt;/a&gt;. The key features of the test involve:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;relatively large number of collections (16k)&lt;/li&gt;
	&lt;li&gt;16 threads each looping through 1k of the 16k collections writing a record and doing a createIndex (on an existing index, not a new index).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Memory outside WT cache is observed to steadily grow over a period of hours in a test on 3.0.5:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/88399/88399_ss.png&quot; width=&quot;800&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;allocated memory outside of WT cache (3rd line) grows steadily.&lt;/li&gt;
	&lt;li&gt;there is a sudden increase corresponding to the known issue with WT journal allocation in 3.0.5.&lt;/li&gt;
	&lt;li&gt;aside from that sudden jump the growth in non-cache memory is linear over time.&lt;/li&gt;
	&lt;li&gt;number of cached cursors is also growing and that is a known potential user of memory, but the shape of the growth curve for number of cursors does not match the linear shape of the growth of non-cached memory.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Memory profiling using tcmalloc HEAPPROFILE shows the following as a candidate for the culprit: various allocations within in __checkpoint_worker (labeled &quot;A&quot; below) steadily grow, and account for about 1.5 GB of non-cache memory by the end of this run:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/88400/88400_heap.png&quot; width=&quot;500&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

</description>
                <environment></environment>
        <key id="228026">SERVER-20248</key>
            <summary>Memory growth in __wt_session_get_btree in __checkpoint_worker under WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="michael.cahill@mongodb.com">Michael Cahill</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>WTmem</label>
                    </labels>
                <created>Tue, 1 Sep 2015 20:16:06 +0000</created>
                <updated>Mon, 11 Jan 2016 20:10:45 +0000</updated>
                            <resolved>Thu, 3 Sep 2015 13:15:31 +0000</resolved>
                                                    <fixVersion>3.0.6</fixVersion>
                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1021482" author="bruce.lucas@10gen.com" created="Thu, 3 Sep 2015 13:14:18 +0000"  >&lt;p&gt;After a 5+ hour run we can confirm no memory growth on 3.0.6:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/88644/88644_mem.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;In the last two rows we see data handles being created and freed by checkpoints.&lt;/li&gt;
	&lt;li&gt;The numbers are consistent with 48k handles existing for the duration of the test for the 48k tables (16k collections + 32k indexes), and an additional 48k handles being created and destroyed on each checkpoint.&lt;/li&gt;
	&lt;li&gt;While memory isn&apos;t growing, we&apos;re seeing significant constant memory usage outside the cache on this test:
	&lt;ul&gt;
		&lt;li&gt;virtual memory is about 5 GB vs allocated memory of about 3.5 GB. This suggests significant fragmentation accounting for about 1.5 GB of extra memory.&lt;/li&gt;
		&lt;li&gt;allocated memory outside the cache is about 1.5 GB; I imagine this is largely the 48k data handles?&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1020970" author="bruce.lucas@10gen.com" created="Wed, 2 Sep 2015 19:15:57 +0000"  >&lt;p&gt;One-hour run with close_idle_time=300 (five minutes) shows no memory growth:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/88580/88580_close_idle_time%3D300.png&quot; width=&quot;650&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This supports the theory that the issue is an accumulation of handles created by the checkpoint.&lt;/p&gt;

&lt;p&gt;There is still a large amount of memory outside the cache. 800 MB is log slot buffer (due to the issue with that in 3.0.5), assume the rest is just due to the large number of dhandles open simply due to the 48k tables (16k collections + 32k indexes).&lt;/p&gt;

&lt;p&gt;Next run will be on 3.0.6 to a) confirm issue still exists there and b) look at the new data-handle stats.&lt;/p&gt;</comment>
                            <comment id="1020771" author="bruce.lucas@10gen.com" created="Wed, 2 Sep 2015 16:28:00 +0000"  >&lt;p&gt;A couple improvements to the tooling more clearly show about 3 GB of memory allocated by __conn_dhandle_get within a checkpoint (&quot;A&quot; below), presumably not accounted for in the WT cache, rising linearly over the course of a 3-hour run.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/88550/88550_heap2.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This run had syncdelay set to 5 seconds vs default 60 seconds. That did not clearly increase the rate of memory increase, but I think that&apos;s because checkpoints were taking a very long time so the number of checkpoints was about the same in either case.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="187365">SERVER-17456</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="88580" name="close_idle_time=300.png" size="54899" author="bruce.lucas@mongodb.com" created="Wed, 2 Sep 2015 19:12:27 +0000"/>
                            <attachment id="88400" name="heap.png" size="320766" author="bruce.lucas@mongodb.com" created="Tue, 1 Sep 2015 20:16:06 +0000"/>
                            <attachment id="88550" name="heap2.png" size="164285" author="bruce.lucas@mongodb.com" created="Wed, 2 Sep 2015 16:23:22 +0000"/>
                            <attachment id="88644" name="mem.png" size="135095" author="bruce.lucas@mongodb.com" created="Thu, 3 Sep 2015 13:13:41 +0000"/>
                            <attachment id="88399" name="ss.png" size="82593" author="bruce.lucas@mongodb.com" created="Tue, 1 Sep 2015 20:16:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 3 Sep 2015 00:21:43 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 23 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 23 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkvp3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hscv2n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsfp6f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>