<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:33:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-76621] Thread pool task executor can cause memory leak when handling exhaust command.</title>
                <link>https://jira.mongodb.org/browse/SERVER-76621</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Thread pool task executor when it handles exhaust commands can leak resources. We noticed the leakage with SingleServerDiscoveryMonitor which is the only consumer at this moment which uses thread pool task executor to schedule exhaust commands . Below is the racy sequence with exhaust command workflow.&lt;br/&gt;
&#160;&lt;br/&gt;
1.&#160; &lt;b&gt;Thread&#160;1:&lt;/b&gt; Starts processing the streamable hello cmd response&#160; and runs &lt;a href=&quot;https://github.com/10gen/mongo/blob/c35bad3b048e8d885bf0b7517aacd2349ea81d14/src/mongo/executor/thread_pool_task_executor.cpp#L842&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;line 842,&lt;/a&gt; and returns true (i.e, &lt;tt&gt;cbState&lt;/tt&gt; is neither canceled nor finished at this point);&#160; and the thread pauses.&lt;/p&gt;

&lt;p&gt;2.&#160;&lt;b&gt;Thread 2:&lt;/b&gt;&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ecc1631b5b299b0753909f7abf874008e5e51e7e/src/mongo/executor/thread_pool_task_executor.cpp#L484&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Cancels the exhaust command&lt;/a&gt; which will cancel the&#160;&lt;tt&gt;cbState&lt;/tt&gt;&#160;and&#160;&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/c35bad3b048e8d885bf0b7517aacd2349ea81d14/src/mongo/executor/thread_pool_task_executor.cpp#L735-L739&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;resets the callback&#160;&lt;/a&gt;in&#160;&lt;tt&gt;cbstate&lt;/tt&gt;&#160; to release any&#160; resources the callback is holding.&lt;/p&gt;

&lt;p&gt;3.&#160;&lt;b&gt;Thread 1:&lt;/b&gt; Thread resumes, then acquires the executor mutex lock and sets the callback in&#160; &lt;tt&gt;cbState&lt;/tt&gt;&#160;&#160; back to the&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/c35bad3b048e8d885bf0b7517aacd2349ea81d14/src/mongo/executor/thread_pool_task_executor.cpp#L844&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;original callback function&lt;/a&gt;. And, this can cause memory leak.&lt;/p&gt;

&lt;p&gt;In the BF, we were ending up with indirect leak due to this circular reference: &#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/client/server_discovery_monitor.h#L39&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SingleServerDiscoveryMonitor&lt;/a&gt;&#160;holds a reference to&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/client/server_discovery_monitor.h#L114&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;TaskExecutor::CallbackHandle&lt;/a&gt;&#160;and which in-turn holds a reference to&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/executor/task_executor.h#L522&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ThreadPoolTaskExecutor::CallbackState&#160;&lt;/a&gt;&#160;and that contains&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/executor/thread_pool_task_executor.cpp#L96&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;CallbackFn&lt;/a&gt;&#160;which holds a reference to&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/client/server_discovery_monitor.cpp#L259&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SingleServerDiscoveryMonitor&lt;/a&gt;.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2326417">SERVER-76621</key>
            <summary>Thread pool task executor can cause memory leak when handling exhaust command.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="jason.chan@mongodb.com">Jason Chan</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Thu, 27 Apr 2023 19:31:14 +0000</created>
                <updated>Sun, 29 Oct 2023 21:22:23 +0000</updated>
                            <resolved>Fri, 2 Jun 2023 13:49:37 +0000</resolved>
                                                    <fixVersion>7.1.0-rc0</fixVersion>
                    <fixVersion>6.0.7</fixVersion>
                    <fixVersion>7.0.0-rc3</fixVersion>
                                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="5482096" author="xgen-internal-githook" created="Wed, 7 Jun 2023 17:49:32 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Jason Chan&apos;, &apos;email&apos;: &apos;jason.chan@mongodb.com&apos;, &apos;username&apos;: &apos;jasonjhchan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-76621&quot; title=&quot;Thread pool task executor can cause memory leak when handling exhaust command.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-76621&quot;&gt;&lt;del&gt;SERVER-76621&lt;/del&gt;&lt;/a&gt; Clarify ThreadPoolTaskExecutor mutex acquisition rules&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 00ae304e83e4e2091262fcda04fa2bc8b8075bbd)&lt;br/&gt;
Branch: v6.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b35e368f862fed371fdad1f3cd75920c82c37547&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b35e368f862fed371fdad1f3cd75920c82c37547&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5476436" author="xgen-internal-githook" created="Mon, 5 Jun 2023 21:52:02 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Jason Chan&apos;, &apos;email&apos;: &apos;jason.chan@mongodb.com&apos;, &apos;username&apos;: &apos;jasonjhchan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-76621&quot; title=&quot;Thread pool task executor can cause memory leak when handling exhaust command.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-76621&quot;&gt;&lt;del&gt;SERVER-76621&lt;/del&gt;&lt;/a&gt; Clarify ThreadPoolTaskExecutor mutex acquisition rules&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 00ae304e83e4e2091262fcda04fa2bc8b8075bbd)&lt;br/&gt;
Branch: v7.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9d961b95ce79500808bbe19131142d8a2d6b60bc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9d961b95ce79500808bbe19131142d8a2d6b60bc&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5469134" author="xgen-internal-githook" created="Thu, 1 Jun 2023 23:08:01 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Jason Chan&apos;, &apos;email&apos;: &apos;jason.chan@mongodb.com&apos;, &apos;username&apos;: &apos;jasonjhchan&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-76621&quot; title=&quot;Thread pool task executor can cause memory leak when handling exhaust command.&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-76621&quot;&gt;&lt;del&gt;SERVER-76621&lt;/del&gt;&lt;/a&gt; Clarify ThreadPoolTaskExecutor mutex acquisition rules&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/00ae304e83e4e2091262fcda04fa2bc8b8075bbd&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/00ae304e83e4e2091262fcda04fa2bc8b8075bbd&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5382636" author="suganthi.mani" created="Thu, 27 Apr 2023 20:14:00 +0000"  >&lt;p&gt;Two ways to fix the problem:&#160;&lt;/p&gt;

&lt;p&gt;1) The fix is to run &lt;a href=&quot;https://github.com/10gen/mongo/blob/c35bad3b048e8d885bf0b7517aacd2349ea81d14/src/mongo/executor/thread_pool_task_executor.cpp#L842-L844&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; block in a single critical section&lt;/p&gt;

&lt;p&gt;2) Break the cycle, &lt;a href=&quot;https://github.com/10gen/mongo/blob/ab714adf0e71005e00ed42367f41ba864b4e6170/src/mongo/client/server_discovery_monitor.cpp#L388-L393&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;by reseting the&#160; handles&lt;/a&gt; after shutting down the singleServerDiscoveryMonitor.&lt;/p&gt;

&lt;p&gt;I prefer fix#1 as I feel fix#2 is sub-optimal as there are other ways to cancel the requests scheduled in thread pool task executor.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;To be noted, we can also get into similar leak problem when &lt;a href=&quot;https://github.com/10gen/mongo/blob/ecc1631b5b299b0753909f7abf874008e5e51e7e/src/mongo/executor/thread_pool_task_executor.cpp#L665&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;callback to last response in the exhaust cursor &lt;/a&gt;&#160;races with &lt;a href=&quot;https://github.com/10gen/mongo/blob/ecc1631b5b299b0753909f7abf874008e5e51e7e/src/mongo/executor/thread_pool_task_executor.cpp#L842-L845&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;callback to previous response&lt;/a&gt;.&lt;/b&gt;  I see lot of fragile places in thread pool task executor, example, if blocks like &lt;a href=&quot;https://github.com/10gen/mongo/blob/ecc1631b5b299b0753909f7abf874008e5e51e7e/src/mongo/executor/thread_pool_task_executor.cpp#L831&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt;  where we may do some work thinking that if check is still valid.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25578"><![CDATA[v7.0]]></customfieldvalue>
    <customfieldvalue key="23470"><![CDATA[v6.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 15 May 2023 17:30:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        35 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            35 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i26q0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1p0p4:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="7349">Service Arch 2023-05-15</customfieldvalue>
    <customfieldvalue id="7350">Service Arch 2023-05-29</customfieldvalue>
    <customfieldvalue id="7351">Service Arch 2023-06-12</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i26c5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>