<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:10:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-6012] Seg Fault, Invalid access at address, Logstream::get called in uninitialized state, on 64bit mongodb 2.0.6</title>
                <link>https://jira.mongodb.org/browse/SERVER-6012</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello.  I&apos;m hoping you can help me figure out why my mongod instance crashes when I execute a certain query.  When I make the below query, I consistently kill the running mongod instance and receive same error message. &lt;/p&gt;

&lt;p&gt;I make the below query to a collection containing 170,941,610 documents.  The collection has an index on the field, &quot;transformed&quot;.&lt;/p&gt;

&lt;p&gt;Tue Jun  5 22:40:25 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn933&amp;#93;&lt;/span&gt; query &amp;lt;DB&amp;gt;.stats_cumulative_stats query: &lt;/p&gt;
{ transformed: false }
&lt;p&gt; ntoreturn:500000 nscanned:58 255 nreturned:58254 reslen:4194308 607ms&lt;/p&gt;


&lt;p&gt;The log shows this error:&lt;/p&gt;


&lt;p&gt;Tue Jun  5 22:40:33 Invalid access at address: 0x3d27000&lt;/p&gt;

&lt;p&gt;Tue Jun  5 22:40:33 Got signal: 11 (Segmentation fault).&lt;/p&gt;

&lt;p&gt;Tue Jun  5 22:40:33 Backtrace:&lt;br/&gt;
0xa95ce9 0xa9640c 0x7fbcbcb5dc60 0x7fbcbc0a3885 0x571617 0x911b3f 0x941b2c 0x885a24 0x88e6da 0xaa0bc8 0x6389f7 0x7fbcbcb54d8c 0x7fbcbc0fec2d.&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3a9) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa95ce9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x22c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa9640c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fbcbcb5dc60&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(memcpy+0x425) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fbcbc0a3885&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZNK5mongo10Projection9transformERKNS_7BSONObjERNS_14BSONObjBuilderE+0x147) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x571617&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo22fillQueryResultFromObjERNS_11_BufBuilderINS_16TrivialAllocatorEEEPNS_10ProjectionERKNS_7BSONObjEPNS_7DiskLocE+0xaf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x911b3f&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14processGetMoreEPKcixRNS_5CurOpEiRb+0x87c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x941b2c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15receivedGetMoreERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x1b4) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x885a24&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xf5a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88e6da&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa0bc8&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6389f7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fbcbcb54d8c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fbcbc0fec2d&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Logstream::get called in uninitialized state&lt;br/&gt;
Tue Jun  5 22:40:33 ERROR: Client::~Client _context should be null but is not; client:conn&lt;br/&gt;
Logstream::get called in uninitialized state&lt;br/&gt;
Tue Jun  5 22:40:33 ERROR: Client::shutdown not called: conn&lt;/p&gt;


&lt;p&gt;&amp;#8212;&lt;/p&gt;


&lt;p&gt;To recover the server, I follow these steps (my db is journaled):  First, I start it up with /etc/init.d/mongodb start.  At this point, I cannot log in via mongo client (ie mongo --host &amp;lt;HOST&amp;gt;) returns &quot;connect failed&quot;.  This is the error message: &lt;/p&gt;

&lt;p&gt;MongoDB shell version: 2.0.4&lt;br/&gt;
connecting to: &amp;lt;HOST&amp;gt;:27017/&amp;lt;DB&amp;gt;&lt;br/&gt;
Tue Jun  5 19:03:12 Error: couldn&apos;t connect to server &amp;lt;HOST&amp;gt;:&amp;lt;PORT&amp;gt; shell/mongo.js:86&lt;br/&gt;
exception: connect failed&lt;/p&gt;

&lt;p&gt;Then, I restart server via /etc/init.d/mongodb restart and I can log in as usual via the mongo client.&lt;/p&gt;</description>
                <environment>$ uname -a&lt;br/&gt;
Linux &amp;lt;NODENAME_HIDDEN&amp;gt; #50-Ubuntu SMP Mon Sep 12 21:51:23 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;mongod --version&lt;br/&gt;
db version v2.0.6, pdfile version 4.5&lt;br/&gt;
Tue Jun  5 23:37:53 git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;mongod --version&lt;br/&gt;
db version v2.0.6, pdfile version 4.5&lt;br/&gt;
Tue Jun  5 23:37:53 git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc&lt;br/&gt;
</environment>
        <key id="40507">SERVER-6012</key>
            <summary>Seg Fault, Invalid access at address, Logstream::get called in uninitialized state, on 64bit mongodb 2.0.6</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="milkie@mongodb.com">Eric Milkie</assignee>
                                    <reporter username="adgaudio">Alex Gaudio</reporter>
                        <labels>
                            <label>crash</label>
                    </labels>
                <created>Tue, 5 Jun 2012 23:39:33 +0000</created>
                <updated>Wed, 15 Aug 2012 14:24:07 +0000</updated>
                            <resolved>Tue, 7 Aug 2012 21:25:11 +0000</resolved>
                                    <version>2.0.6</version>
                                                    <component>Internal Code</component>
                    <component>Logging</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="151304" author="milkie" created="Tue, 7 Aug 2012 21:25:11 +0000"  >&lt;p&gt;Closing as incomplete; we will watch for this crash to happen again.&lt;/p&gt;</comment>
                            <comment id="136775" author="milkie" created="Tue, 26 Jun 2012 19:04:12 +0000"  >&lt;p&gt;Hi Alex,&lt;br/&gt;
Without any log data, it&apos;s hard to tell exactly what happened, and I can&apos;t really diagnose the problem without something further to go on.&lt;/p&gt;</comment>
                            <comment id="136485" author="adgaudio" created="Tue, 26 Jun 2012 01:18:01 +0000"  >&lt;p&gt;Hey Eric, &lt;/p&gt;

&lt;p&gt;Thanks for your response - I repaired the collection and am operating normally.  &lt;/p&gt;

&lt;p&gt;At this point, I&apos;m confused why the server crashed and corrupted my data when the log partition filled up, given that the db data and logs are in two separate partitions.  I wouldn&apos;t be surprised if there&apos;s a bug or two that contributed to the problem.&lt;/p&gt;

&lt;p&gt;Also, a possible future feature: Given mongo&apos;s instability when it runs out of disk space mid-write, it may be helpful to add some sort of corruption protection or ability to shutdown the server instance when the server log approaches max disk usage.&lt;/p&gt;

&lt;p&gt;Thanks for your help.  Alex&lt;/p&gt;</comment>
                            <comment id="133000" author="milkie" created="Fri, 15 Jun 2012 00:24:07 +0000"  >&lt;p&gt;Let me see if I can surmise what happened here.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$ sudo mv /var/logs/mongodb/{mongodb.log,mongodb.log2} ; sudo rm -rf /var/logs/mongodb/mongodb.log2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$ sudo -u mongodb /usr/bin/mongod --port 27017 --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb2.log --auth # --auth may not have been there&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The move of the file does not affect the open file handle, so mongod keeps attempting to extend the file (but can&apos;t).&lt;br/&gt;
The delete of the file does not affect the open file handle either, although it does make it impossible to look up the file in the directory again.  mongod keeps attempting to extend the file.&lt;br/&gt;
When you attempted to start another mongod, you give it the same log path, which writes a new file in that location since the directory entry was marked as deleted in the prior command.  There are now two mongod&apos;s attempting to write to the disk.  The new mongod hasn&apos;t yet attempted to open the network port nor the lock file on disk, so it doesn&apos;t know that the port is already in use and that the data directory is already in use.&lt;br/&gt;
Eventually, something must have freed up disk space enough for both instances of mongod to exit.&lt;/p&gt;


&lt;p&gt;At this point, it would seem that your datafiles are corrupt.  The usual procedures for recovery apply here: restore from backup, or from another node in a replica set, or, failing that, running repair &amp;#8211; which might lose any number of records and which will only fix structural corruption &amp;#8211; data corruption could remain unless you have a way of checking for it yourself.&lt;/p&gt;</comment>
                            <comment id="130626" author="adgaudio" created="Mon, 11 Jun 2012 22:45:48 +0000"  >&lt;p&gt;&amp;gt;&amp;gt; Did you delete the log file while the server was still running?&lt;br/&gt;
Yea, see below.  The delete-pending fact is interesting and may explain some of my response to your third question&lt;/p&gt;

&lt;p&gt;&amp;gt;&amp;gt; What do you mean specifically by &quot;forked another mongod instance&quot; &lt;/p&gt;

&lt;p&gt;Sorry, I mis-spoke a little.  Here&apos;s basically a replay of what happened when I deleted the log file and managed to restart the server:&lt;br/&gt;
$ sudo /etc/init.d/mongodb stop # hangs ... &lt;/p&gt;


&lt;p&gt;$ df -h&lt;br/&gt;
    /dev/xvda1            7.9G  7.9G  0  100% &lt;/p&gt;

&lt;p&gt;$ ps -ef|grep mongo&lt;br/&gt;
    mongodb  17665     1  0 Jun05 ?        00:46:01 /usr/bin/mongod --port 27017 --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb.log --auth&lt;/p&gt;

&lt;p&gt;$ sudo mv /var/logs/mongodb/&lt;/p&gt;
{mongodb.log,mongodb.log2}
&lt;p&gt; ; sudo rm -rf /var/logs/mongodb/mongodb.log2&lt;br/&gt;
$ sudo -u mongodb /usr/bin/mongod --port 27017 --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongodb2.log --auth # --auth may not have been there&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;ps -ef now shows 2 mongod instances&lt;br/&gt;
$ sudo /etc/init.d/mongodb stop # nothing in this output indicated an improper shutdown&lt;/li&gt;
	&lt;li&gt;ps -ef now shows 0 mongod instances&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&amp;gt;&amp;gt; Do you have a log from the server where you attempted to start it after the failure and yet couldn&apos;t connect?&lt;/p&gt;

&lt;p&gt;Don&apos;t have that log anymore, but if I remember correctly, I don&apos;t think the server logged anything (which suggests that perhaps the server log was still somehow &quot;delete-pending&quot;?).  I just tried to regenerate the situation where I couldn&apos;t connect my mongo client to the server, but can&apos;t reproduce that error anymore.  Maybe I can&apos;t reproduce because I removed indexes on this collection last Friday.  &lt;/p&gt;

&lt;p&gt;That said, while my server isn&apos;t freezing up since deleting the indexes, I still do get this message about corrupt data from a mongo shell:&lt;br/&gt;
&amp;gt; db.stats_cumulative_stats.find(&lt;/p&gt;
{parsed: false}
&lt;p&gt;)                 &#9474;                                                                 &lt;br/&gt;
                                                                  &#9474;                                                                 &lt;br/&gt;
error: &lt;/p&gt;
{ &quot;$err&quot; : &quot;BSONElement: bad type -48&quot;, &quot;code&quot; : 10320 }
&lt;p&gt;   2;;bW9uZ28gLS1ob3N0IGVjMi0xMDctMjItNjEtODIuY29tcHV0ZS0xLmFtYXpvCk1&amp;gt;          &lt;/p&gt;


&lt;p&gt;The corresponding log (similar to initial log I posted):&lt;/p&gt;

&lt;p&gt;&amp;lt;&amp;lt; line about conn1 authentication&amp;gt;&amp;gt;&amp;gt;&lt;br/&gt;
Mon Jun 11 22:17:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:836 virt:601889 mapped:300862&lt;br/&gt;
Mon Jun 11 22:18:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:3335 virt:601889 mapped:300862&lt;br/&gt;
Mon Jun 11 22:20:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:3952 virt:601889 mapped:300862&lt;br/&gt;
Mon Jun 11 22:20:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1&amp;#93;&lt;/span&gt; Assertion: 10320:BSONElement: bad type -48&lt;br/&gt;
0x5848a2 0x507cdb 0x664e11 0x666afd 0x86e5bd 0x86e7f8 0x9506c2 0x8c9616 0x8d9700 0x8d9d56 0x8dd66e 0x8df083 0x8e0357 0x943295 0x8891&lt;br/&gt;
07 0x88dcd9 0xaa0bc8 0x6389f7 0x7f2f23776d8c 0x7f2f22d20c2d &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x112) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5848a2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZNK5mongo11BSONElement4sizeEv+0x1cb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x507cdb&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7Matcher13matchesDottedEPKcRKNS_11BSONElementERKNS_7BSONObjEiRKNS_14ElementMatcherEbPNS_12MatchDetailsE+0x&lt;br/&gt;
1b01) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x664e11&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7Matcher7matchesERKNS_7BSONObjEPNS_12MatchDetailsE+0xdd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x666afd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher7matchesERKNS_7BSONObjERKNS_7DiskLocEPNS_12MatchDetailsEb+0xcd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86e5bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19CoveredIndexMatcher14matchesCurrentEPNS_6CursorEPNS_12MatchDetailsE+0xa8) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x86e7f8&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x262) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9506c2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8c9616&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8d9700&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8d9d56&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8dd66e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x523) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8df083&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8e0357&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(&lt;em&gt;ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1&lt;/em&gt;+0xfa5) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x943295&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889107&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88dcd9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x78) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa0bc8&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6389f7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2f23776d8c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f2f22d20c2d&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Jun 11 22:20:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1&amp;#93;&lt;/span&gt; assertion 10320 BSONElement: bad type -48 ns:&amp;lt;DB&amp;gt;.stats_cumulative_stats query:&lt;/p&gt;
{ parsed: false
 }
&lt;p&gt;Mon Jun 11 22:20:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1&amp;#93;&lt;/span&gt; query &amp;lt;DB&amp;gt;.stats_cumulative_stats query: &lt;/p&gt;
{ parsed: false }
&lt;p&gt; exception: BSONElement: bad type -4&lt;br/&gt;
8 code:10320 reslen:71 231049ms&lt;br/&gt;
Mon Jun 11 22:22:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1&amp;#93;&lt;/span&gt; end connection &amp;lt;IP&amp;gt;:48274&lt;br/&gt;
Mon Jun 11 22:23:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from &amp;lt;IP&amp;gt;:48332 #2&lt;br/&gt;
Mon Jun 11 22:25:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:3940 virt:601889 mapped:300862&lt;br/&gt;
Mon Jun 11 22:30:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;clientcursormon&amp;#93;&lt;/span&gt; mem (MB) res:3940 virt:601889 mapped:300862&lt;/p&gt;


&lt;p&gt;----------------------&lt;br/&gt;
So at this point, I&apos;d say there were/are 2 problems:&lt;/p&gt;

&lt;p&gt;1. partition filled up mid-write and corrupted data.  &lt;br/&gt;
2. indexes on that table were broken and caused a query using the index to crash the database.&lt;/p&gt;</comment>
                            <comment id="130255" author="milkie" created="Mon, 11 Jun 2012 13:15:53 +0000"  >&lt;p&gt;Hi Alex.&lt;br/&gt;
Did you delete the log file while the server was still running?  This would have just made the file delete-pending, and the server would have continued try to extend the log file. &lt;br/&gt;
What do you mean specifically by &quot;forked another mongod instance&quot; &amp;#8211; you pointed the new instance at a different data directory?&lt;br/&gt;
Do you have a log from the server where you attempted to start it after the failure and yet couldn&apos;t connect?&lt;/p&gt;</comment>
                            <comment id="130037" author="adgaudio" created="Mon, 11 Jun 2012 03:12:09 +0000"  >&lt;p&gt;Hi Eric, thanks for taking this one.  I should also add that this might be due to an event that occurred a few days before I filed this bug: namely, our log partition filled up and locked the mongo server.  After I deleted the log file, the mongod instance wouldn&apos;t shut down (ie &quot;/$ etc/init.d/mongodb stop&quot; would just hang).  I didn&apos;t explicitly kill -9 the mongod instance, but instead forked another mongod instance and used /etc/init.d/mongodb stop to shut down both.  &lt;/p&gt;

&lt;p&gt;I also didn&apos;t mention that we are not replicated or sharded.  &lt;/p&gt;

&lt;p&gt;Hope that helps! Thanks, Alex&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 11 Jun 2012 13:15:53 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 28 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 28 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>adgaudio</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hro0xr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrir53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23050</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht08br:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>