<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:06:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4607] cases of memory corruption in mongod and mongos on exit</title>
                <link>https://jira.mongodb.org/browse/SERVER-4607</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Analysis below is relatively speculative, and I haven&apos;t proven any of it via testing.  Hopefully this ticket is at least useful as a survey of cases where we&apos;ve seen memory corruption during shutdown, often as a cascade failure after an earlier failure triggered the shutdown itself.&lt;/p&gt;

&lt;p&gt;It looks like memory corruption can be triggered after an unclean shutdown and potentially can occur after a clean shutdown too.  From the examples I&apos;ve seen it kind of looks like double frees are occurring because global objects with members that manage their own heap memory are destroyed as the process is exiting, and then the heap memory of these members are freed again as a result of actions taken by a thread that&apos;s still running.&lt;/p&gt;

&lt;p&gt;Here are some historical jira cases where I&apos;ve seen this occur:&lt;/p&gt;

&lt;p&gt;1)&lt;/p&gt;

&lt;p&gt;When Top::cloneMap called by SnapshotData::takeSnapshot.  Potentially caused by the global statsSnapshots variable getting destroyed, and then one of its SnapshotData also getting destroyed but having takeSnapshot called on it and trying to reassign its _usage map.  Potentially the _usage map was already destroyed, and its heap memory was freed, but the reassignment attempts to free the heap memory again.  Stack traces that might be related to this found in:&lt;/p&gt;

&lt;p&gt;FREE-3600&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-2695&quot; title=&quot;Server crash during restore - cannot allocate memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-2695&quot;&gt;&lt;del&gt;SERVER-2695&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4190&quot; title=&quot;SEGFAULT doing query&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4190&quot;&gt;&lt;del&gt;SERVER-4190&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It looks like SnapshotThread::run checks inShutdown(), but a shutdown occurs after after the inShutdown() check but before or during the call to takeSnapshot() the same double free might occur as in the unclean shutdown cases.&lt;/p&gt;

&lt;p&gt;2)&lt;/p&gt;

&lt;p&gt;Maybe when an an immediate exit occurs during a shutdown, global objects required for shutdown may be destroyed while still in use?&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3869&quot; title=&quot;mongod 2.0.0 crash &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3869&quot;&gt;&lt;del&gt;SERVER-3869&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;3)&lt;/p&gt;

&lt;p&gt;Here it kind of looks like there is a global freed or left in a bad state and then another exit call attempts to free it again.&lt;/p&gt;

&lt;p&gt;CS-501&lt;/p&gt;

&lt;p&gt;4)&lt;/p&gt;

&lt;p&gt;Mystery failure after a clean shutdown&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-414&quot; title=&quot;Memory corruption on initial pre-allocation&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-414&quot;&gt;&lt;del&gt;SERVER-414&lt;/del&gt;&lt;/a&gt; (very old mongo version)&lt;/p&gt;

&lt;p&gt;5)&lt;/p&gt;

&lt;p&gt;On mongos these failures may potentially have occurred in similar situations:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3082&quot; title=&quot;mongos crash on shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3082&quot;&gt;&lt;del&gt;SERVER-3082&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4367&quot; title=&quot;segfault in mongos&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4367&quot;&gt;&lt;del&gt;SERVER-4367&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
CS-1903&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-2930&quot; title=&quot;Mongos crashes during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-2930&quot;&gt;&lt;del&gt;SERVER-2930&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
FREE-3696&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4576&quot; title=&quot;mongos stack traces after bind failure&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4576&quot;&gt;&lt;del&gt;SERVER-4576&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;I would recommend that we do a closer examination of 2-5 above, do an audit for additional cases, and then fix all known cases.&lt;/p&gt;</description>
                <environment></environment>
        <key id="27624">SERVER-4607</key>
            <summary>cases of memory corruption in mongod and mongos on exit</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="aaron">Aaron Staple</reporter>
                        <labels>
                    </labels>
                <created>Tue, 3 Jan 2012 22:27:24 +0000</created>
                <updated>Tue, 6 Dec 2022 05:37:53 +0000</updated>
                            <resolved>Tue, 15 Nov 2016 20:49:24 +0000</resolved>
                                                                    <component>Internal Code</component>
                                        <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                        <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 7 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 7 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>aaron</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrohtb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrft1z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4890</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0q7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>