<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:09:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-5497] mongod: double free or corruption </title>
                <link>https://jira.mongodb.org/browse/SERVER-5497</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;mongod outputs *** glibc detected *** mongod: double free or corruption (!prev): 0x00002ac5c40af3b0 ***&lt;br/&gt;
in the logs.&lt;/p&gt;

&lt;p&gt;The replicaset seems to have stopped replication for a long time (that may be another issue). &lt;br/&gt;
Primary started output weird errors &quot;double free or corruption&quot;. process was still alive (until we kill -9 it) but refuses any connections (shell or with other replicasets members).&lt;/p&gt;
</description>
                <environment>linux x86_64   2.6.18-274.3.1.el5.centos.plus</environment>
        <key id="35443">SERVER-5497</key>
            <summary>mongod: double free or corruption </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="dan@mongodb.com">Daniel Pasette</assignee>
                                    <reporter username="kamaradclimber">Gr&#233;goire Seux</reporter>
                        <labels>
                    </labels>
                <created>Wed, 4 Apr 2012 09:01:18 +0000</created>
                <updated>Wed, 10 Dec 2014 23:18:26 +0000</updated>
                            <resolved>Thu, 25 Jul 2013 13:15:13 +0000</resolved>
                                    <version>2.0.2</version>
                                                    <component>Stability</component>
                                        <votes>2</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="309813" author="dan@10gen.com" created="Tue, 9 Apr 2013 23:36:15 +0000"  >&lt;p&gt;This issue got lost in the shuffle.  Is this still an active problem for any watchers? &lt;/p&gt;</comment>
                            <comment id="141201" author="kamaradclimber" created="Tue, 10 Jul 2012 07:17:04 +0000"  >&lt;p&gt;My setup is not the same as Theo, Andy or Petr.&lt;br/&gt;
We ran (at that time) 2.0.2 normal version.&lt;/p&gt;</comment>
                            <comment id="140920" author="schwerin" created="Mon, 9 Jul 2012 16:42:24 +0000"  >&lt;p&gt;So, your logs are from the 2.0.1 regular version?  I&apos;ll take another look at the symbols.&lt;/p&gt;</comment>
                            <comment id="140717" author="kamaradclimber" created="Mon, 9 Jul 2012 06:32:46 +0000"  >&lt;p&gt;we run on centos5 indeed but not using the legacy-static version. Could it come from somewhere else ?&lt;/p&gt;</comment>
                            <comment id="140392" author="schwerin" created="Fri, 6 Jul 2012 17:01:55 +0000"  >&lt;p&gt;Did some more digging.  If you can, you should stop running the legacy-static version of mongo.  It looks like you&apos;re on Centos5, which supports the regular build of mongo.  The legacy-static version tries to statically link against a number of libraries that aren&apos;t super safe to statically link against.&lt;/p&gt;

&lt;p&gt;I was hoping to use the debug symbols to provide a more detailed analysis, but it seems that we don&apos;t have debugging symbols available for the 2.0.2 legacy static build &amp;#8211; some kind of gaf during the release process for versions 2.0.0 to 2.0.4.&lt;/p&gt;</comment>
                            <comment id="133282" author="thesaint" created="Fri, 15 Jun 2012 14:11:22 +0000"  >&lt;p&gt;Also wondering if there is any progress on this. We see this issue fairly regularly.&lt;/p&gt;</comment>
                            <comment id="133273" author="kamaradclimber" created="Fri, 15 Jun 2012 13:50:45 +0000"  >&lt;p&gt;Is there progress on this issue ?&lt;/p&gt;</comment>
                            <comment id="115699" author="thesaint" created="Wed, 2 May 2012 18:53:45 +0000"  >&lt;p&gt;Here is end of the log from another node that crashed:&lt;/p&gt;

&lt;p&gt;Wed May  2 04:02:01 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.199.144.35:33373 #40442&lt;br/&gt;
Wed May  2 04:02:10 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; repl: old cursor isDead, will initiate a new one&lt;br/&gt;
Wed May  2 04:02:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet syncing to: ahost.adomain.com:27017&lt;br/&gt;
Wed May  2 04:02:14 Invalid access at address: 0&lt;/p&gt;

&lt;p&gt;Wed May  2 04:02:15 Got signal: 11 (Segmentation fault).&lt;/p&gt;

&lt;p&gt;Wed May  2 04:02:16 Backtrace:&lt;br/&gt;
0x986fb9 0x987590 0x9a3810 0x3ec666afa2 0x2aaaaaab933f 0x2aaaac08b138 0xb86029 &lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x986fb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x987590&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9a3810&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib64/libc.so.6(fgets_unlocked+0x22) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x3ec666afa2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib64/libnss_files.so.2 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x2aaaaaab933f&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x2aaaac08b138&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb86029&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;
		&lt;ul&gt;
			&lt;li&gt;glibc detected *** ./mongod: double free or corruption (!prev): 0x00002aacfc060a90 ***&lt;br/&gt;
======= Backtrace: =========&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0xa71c1b&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0xa75cc6&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x594641&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x59604f&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x590dc2&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x591021&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x592785&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x4768d2&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x477744&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0xa09f69&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0x99f9ad&amp;#93;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;0xa8f549&amp;#93;&lt;/span&gt;&lt;br/&gt;
======= Memory map: ========&lt;br/&gt;
00400000-00c97000 r-xp 00000000 08:01 2261002                            /u00/mongo1/bin/mongod&lt;br/&gt;
00e97000-00eb8000 rw-p 00897000 08:01 2261002                            /u00/mongo1/bin/mongod&lt;br/&gt;
00eb8000-018d9000 rw-p 00eb8000 00:00 0 &lt;br/&gt;
0dd12000-0de8b000 rw-p 0dd12000 00:00 0                                  &lt;span class=&quot;error&quot;&gt;&amp;#91;heap&amp;#93;&lt;/span&gt;&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="115694" author="thesaint" created="Wed, 2 May 2012 18:46:09 +0000"  >&lt;p&gt;Also seeing the same issue frequently. &lt;/p&gt;

&lt;p&gt;Here is the tail end of the log right before crash:&lt;/p&gt;

&lt;p&gt;Mon Apr 30 17:41:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.199.144.35:47758 #36268&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;
		&lt;ul&gt;
			&lt;li&gt;glibc detected *** ./mongod: double free or corruption (top): 0x00002aad000008c0 ***&lt;br/&gt;
Mon Apr 30 17:41:58 Invalid access at address: 0&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Mon Apr 30 17:41:58 Got signal: 11 (Segmentation fault).&lt;/p&gt;

&lt;p&gt;Mon Apr 30 17:41:58 Backtrace:&lt;br/&gt;
0x986fb9 0x987590 0x9a3810 &lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x986fb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x987590&amp;#93;&lt;/span&gt;&lt;br/&gt;
 &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9a3810&amp;#93;&lt;/span&gt;&lt;/p&gt;


&lt;p&gt;Here is the top of log after start:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;
	&lt;ul&gt;
		&lt;li&gt;
		&lt;ul&gt;
			&lt;li&gt;
			&lt;ul&gt;
				&lt;li&gt;
				&lt;ul&gt;
					&lt;li&gt;SERVER RESTARTED *****&lt;/li&gt;
				&lt;/ul&gt;
				&lt;/li&gt;
			&lt;/ul&gt;
			&lt;/li&gt;
		&lt;/ul&gt;
		&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; MongoDB starting : pid=14551 port=27017 dbpath=/Data/u01/mongo1/data/ 64-bit host=ahost.adomain.com&lt;br/&gt;
Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; db version v2.0.2, pdfile version 4.5&lt;br/&gt;
Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; git version: 514b122d308928517f5841888ceaa4246a7f18e3&lt;br/&gt;
Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 BOOST_LIB_VERSION=1_45&lt;br/&gt;
Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; options: &lt;/p&gt;
{ config: &quot;mongodb.cnf&quot;, dbpath: &quot;/Data/u01/mongo1/data/&quot;, directoryperdb: &quot;true&quot;, fork: &quot;true&quot;, logappend: &quot;true&quot;, logpath: &quot;/u00/mongo1/log/mongodb.log&quot;, maxConns: 20000, port: 27017, profile: 1, replSet: &quot;replicaset_1&quot;, rest: &quot;true&quot;, shardsvr: &quot;true&quot; }
&lt;p&gt;Tue Apr 24 11:23:17 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; journal dir=/Data/u01/mongo1/data/journal&lt;/p&gt;</comment>
                            <comment id="115516" author="pmasopust" created="Wed, 2 May 2012 09:16:31 +0000"  >&lt;p&gt;I have similar issue - mongo is crashing with segfault.&lt;br/&gt;
Server: Linux server48.company.com 2.6.18-194.11.4.el5 #1 SMP Fri Sep 17 04:57:05 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux&lt;br/&gt;
Mongo is 2.0.2 64bit old legacy (i.e. static with all libs included) 3 server replica set.&lt;/p&gt;

&lt;p&gt;See mongo-clean.log with stacktrace&lt;/p&gt;</comment>
                            <comment id="109454" author="kamaradclimber" created="Fri, 13 Apr 2012 08:02:00 +0000"  >&lt;p&gt;I am sorry but we don&quot;t have them anymore. It was started a long time ago and we only keep these logs for 7 days.&lt;/p&gt;</comment>
                            <comment id="106500" author="schwerin" created="Wed, 4 Apr 2012 15:03:20 +0000"  >&lt;p&gt;I&apos;m looking for the startup messages.  The log you sent does not contain them.  Any chance you can start up the same binary again, and send the first 100 lines or so of the log?&lt;/p&gt;</comment>
                            <comment id="106485" author="kamaradclimber" created="Wed, 4 Apr 2012 14:36:30 +0000"  >&lt;p&gt;more complete log&lt;/p&gt;</comment>
                            <comment id="106460" author="schwerin" created="Wed, 4 Apr 2012 14:01:59 +0000"  >&lt;p&gt;Could you send us the first hundred lines or so of the log file from the crashing mongo instance?  Or start up mongod again and send us the first 100 lines of log output?  I want to make sure I grab the right set of debugging symbols to correlate with that stack trace.&lt;/p&gt;</comment>
                            <comment id="106395" author="kamaradclimber" created="Wed, 4 Apr 2012 09:42:40 +0000"  >&lt;p&gt;correction : we use mongo 2.0.1&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="16358" name="mongo-clean.log" size="3889" author="pmasopust" created="Wed, 2 May 2012 09:16:54 +0000"/>
                            <attachment id="15746" name="mongod-shard4.extract.log" size="195805" author="kamaradclimber" created="Wed, 4 Apr 2012 09:01:19 +0000"/>
                            <attachment id="15752" name="mongod-shard4.log" size="2386585" author="kamaradclimber" created="Wed, 4 Apr 2012 14:36:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 4 Apr 2012 14:01:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 45 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 45 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>kamaradclimber</customfieldvalue>
            <customfieldvalue>pmasopust</customfieldvalue>
            <customfieldvalue>thesaint</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hro787:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg8nz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8309</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlj3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>