<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:33:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-54472] Collect additional FTDC metrics during shutdown for replica sets</title>
                <link>https://jira.mongodb.org/browse/SERVER-54472</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-48221&quot; title=&quot;Shut down ftdc after storage engine&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-48221&quot;&gt;&lt;del&gt;SERVER-48221&lt;/del&gt;&lt;/a&gt; we enabled collection of FTDC metrics during WT shutdown; however we limited the metrics collected to system information outside of serverStatus.&lt;/p&gt;

&lt;p&gt;There is some additional system-related information in serverStatus that would be useful to collect, such as tcmalloc and extra_info. This would allow us to diagnose CPU and memory problems during shutdown, and to monitor memory usage including shutdown in our performance tests.&lt;/p&gt;

&lt;p&gt;It should be possible to collect this information since it is system-related and not (logically) dependent on MongoDB-specific data structures.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1620674">SERVER-54472</key>
            <summary>Collect additional FTDC metrics during shutdown for replica sets</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="gregory.wlodarek@mongodb.com">Gregory Wlodarek</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                    </labels>
                <created>Thu, 11 Feb 2021 17:01:58 +0000</created>
                <updated>Sun, 29 Oct 2023 21:57:39 +0000</updated>
                            <resolved>Wed, 31 Mar 2021 01:02:03 +0000</resolved>
                                                    <fixVersion>5.0.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="3692875" author="xgen-internal-githook" created="Wed, 31 Mar 2021 01:01:25 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Gregory Wlodarek&apos;, &apos;email&apos;: &apos;gregory.wlodarek@mongodb.com&apos;, &apos;username&apos;: &apos;GWlodarek&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54472&quot; title=&quot;Collect additional FTDC metrics during shutdown for replica sets&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54472&quot;&gt;&lt;del&gt;SERVER-54472&lt;/del&gt;&lt;/a&gt; Collect additional FTDC metrics during shutdown for replica sets&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ba8681015ac79e0b2f3fa1facdf9ff07c0282322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ba8681015ac79e0b2f3fa1facdf9ff07c0282322&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3692025" author="gregory.wlodarek" created="Tue, 30 Mar 2021 17:54:06 +0000"  >&lt;p&gt;I&apos;ve found the issue. I tested my changes against a standalone node only and not a replica set where there&apos;s a minor difference when shutting down. I&apos;ll use this ticket to fix that.&lt;/p&gt;

&lt;p&gt;I attached sample metrics with the fix.&lt;/p&gt;</comment>
                            <comment id="3635251" author="bruce.lucas@10gen.com" created="Thu, 25 Feb 2021 16:29:27 +0000"  >&lt;p&gt;Maybe, but it looks to me like the presence of the ShutdownInProgress error as the (entire) content of serverStatus during this time suggests to me that the problem doesn&apos;t originate in WT per se.&lt;/p&gt;</comment>
                            <comment id="3635011" author="gregory.wlodarek" created="Thu, 25 Feb 2021 15:18:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;&#160;sorry for getting back to you late. This is an interesting case, I wonder if during the WT shutdown something is preventing serverStatus from running properly. I think this is worth investigating for us.&lt;/p&gt;</comment>
                            <comment id="3624959" author="bruce.lucas@10gen.com" created="Fri, 19 Feb 2021 14:40:50 +0000"  >&lt;p&gt;The content of serverStatus seen in FTDC during shutdown in the above case consists of just the following fields:&lt;/p&gt;

&lt;p&gt;ok: 0&lt;br/&gt;
errmsg: &quot;The server is in quiesce mode and will shut down&quot;&lt;br/&gt;
code: 91&lt;br/&gt;
codeName: &quot;ShutdownInProgress&quot;&lt;br/&gt;
remainingQuiesceTimeMillis: 0&lt;/p&gt;</comment>
                            <comment id="3623894" author="bruce.lucas@10gen.com" created="Thu, 18 Feb 2021 20:28:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregory.wlodarek&quot; class=&quot;user-hover&quot; rel=&quot;gregory.wlodarek&quot;&gt;gregory.wlodarek&lt;/a&gt; if so then I may have misinterpreted what I was seeing in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53708&quot; title=&quot;Excess memory usage during shutdown in durable history tests&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53708&quot;&gt;&lt;del&gt;SERVER-53708&lt;/del&gt;&lt;/a&gt;. From iterations-08.tgz on that ticket:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/300720/300720_ftdc.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;A marks the point where we start shutting down WT. The system memory metrics (from the systemMetrics section of ftdc) shows the increasing memory that WT is using during shutdown, but the tcmalloc and resident memory stats (from serverStatus) are showing as 0.&lt;/p&gt;

&lt;p&gt;This is a little different from your experiment - the delay here is in WT shutdown, whereas in your experiment you introduced a delay after shutdown. I wonder if it&apos;s possible that behavior during WT shutdown is what&apos;s problematic.&lt;/p&gt;

&lt;p&gt;In case it matters, the above is on build&#160;4.9.0-alpha-1214-g54e8728.&lt;/p&gt;</comment>
                            <comment id="3623161" author="gregory.wlodarek" created="Thu, 18 Feb 2021 15:53:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;&#160;I attached screenshots from t2 of metrics we continue to collect after the storage engine shutdown. In the screenshots,&#160;the server was idle for a minute then I hung shutdown after the storage engine shutdown for another minute. Both &lt;tt&gt;tcmalloc&lt;/tt&gt; and &lt;tt&gt;extra_info&lt;/tt&gt; continue to be collected. Am I missing something?&lt;/p&gt;</comment>
                            <comment id="3612008" author="bruce.lucas@10gen.com" created="Thu, 11 Feb 2021 17:08:05 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53708&quot; title=&quot;Excess memory usage during shutdown in durable history tests&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53708&quot;&gt;&lt;del&gt;SERVER-53708&lt;/del&gt;&lt;/a&gt; is an example of the kind of issue we need to be able to detect in testing (without relying on actually hitting OOM as we did in that ticket).&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1586735">SERVER-53708</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1351584">SERVER-48221</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="300644" name="FTDC-1.png" size="315108" author="gregory.wlodarek@mongodb.com" created="Thu, 18 Feb 2021 15:52:16 +0000"/>
                            <attachment id="300643" name="FTDC-2.png" size="328752" author="gregory.wlodarek@mongodb.com" created="Thu, 18 Feb 2021 15:52:16 +0000"/>
                            <attachment id="300642" name="FTDC-3.png" size="360775" author="gregory.wlodarek@mongodb.com" created="Thu, 18 Feb 2021 15:52:16 +0000"/>
                            <attachment id="300720" name="ftdc.png" size="108277" author="bruce.lucas@mongodb.com" created="Thu, 18 Feb 2021 20:28:17 +0000"/>
                            <attachment id="307310" name="metrics.2021-03-30T17-55-08Z-00000" size="66188" author="gregory.wlodarek@mongodb.com" created="Tue, 30 Mar 2021 17:58:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 18 Feb 2021 15:42:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 45 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 45 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>gregory.wlodarek@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyuo9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr41of:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4569">Execution Team 2021-03-22</customfieldvalue>
    <customfieldvalue id="4570">Execution Team 2021-04-05</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyuain:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>