<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:55:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20741] Primary crash after N hours of running as primary</title>
                <link>https://jira.mongodb.org/browse/SERVER-20741</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I manage a sharded cluster for my company. That cluster is used by clients as a free cluster: they provision a db and can use it (with some limitations) with their applications.&lt;/p&gt;

&lt;p&gt;I moved from 2.6 to 3.0.6 a week ago (on Thursday 2015-09-24), and ever since I have this strange behavior: after being elected as primary, a node will last a few hours (between 2 and 5) and then crash.&lt;br/&gt;
The crash is a segmentation fault.&lt;/p&gt;

&lt;p&gt;We have systemd restarting the node automatically, and in the meantime, a new node is elected as primary and run for a few more hours then crashes, and another one is elected primary, etc.&lt;/p&gt;

&lt;p&gt;The cluster is composed of 3 config servers, 3 mongos, and 5 mongod all within a single RS and handling a single shard.&lt;br/&gt;
The 5 mongod are 2 arbiters and 3 data nodes.&lt;br/&gt;
The 3 data nodes are 1 MMAPv1 and 2 wiredTiger.&lt;/p&gt;

&lt;p&gt;All 3 data nodes crash a few hours after being elected master.&lt;/p&gt;

&lt;p&gt;I attached the log of a primary starting 30 seconds before the segfault happens.&lt;/p&gt;

&lt;p&gt;/sys/kernel/mm/transparent_hugepage/defrag does not exist on 2 of the 3 servers, and I set it to &quot;never&quot; on the third one.&lt;/p&gt;</description>
                <environment></environment>
        <key id="232585">SERVER-20741</key>
            <summary>Primary crash after N hours of running as primary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="judu">Julien Durillon</reporter>
                        <labels>
                    </labels>
                <created>Fri, 2 Oct 2015 17:52:37 +0000</created>
                <updated>Mon, 8 Jan 2024 15:22:58 +0000</updated>
                            <resolved>Mon, 12 Oct 2015 21:35:05 +0000</resolved>
                                    <version>3.0.6</version>
                                                    <component>JavaScript</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1157746" author="ramon.fernandez" created="Fri, 29 Jan 2016 13:22:35 +0000"  >&lt;p&gt;For the record, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-22334&quot; title=&quot;mongod crashes on map_reduce call at AdjustAmountOfExternalAllocatedMemory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-22334&quot;&gt;&lt;del&gt;SERVER-22334&lt;/del&gt;&lt;/a&gt; shows how a missing &quot;var&quot; keyword in the JS code could trigger this issue.  &lt;a href=&quot;https://docs.mongodb.org/manual/release-notes/3.2/#spidermonkey-javascript-engine&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;MongoDB 3.2 uses SpiderMonkey as the JavaScript engine&lt;/a&gt; an it handles this case better than V8.&lt;/p&gt;</comment>
                            <comment id="1058478" author="ramon.fernandez" created="Mon, 12 Oct 2015 21:34:56 +0000"  >&lt;p&gt;Thanks for the additional information &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judu&quot; class=&quot;user-hover&quot; rel=&quot;judu&quot;&gt;judu&lt;/a&gt;. Since the issue you describe does not point to a bug in the server I&apos;m going to close this ticket.&lt;/p&gt;

&lt;p&gt;If you need assistance building MongoDB from sources you can post in the &lt;a href=&quot;http://groups.google.com/group/mongodb-dev&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-dev group&lt;/a&gt;; please make sure to provide information about the version of the tools and libraries you&apos;re using. In particular, we don&apos;t yet support compiling with gcc 4.9 or older.&lt;/p&gt;

&lt;p&gt;For user support discussions please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt;. See also our &lt;a href=&quot;https://www.mongodb.org/about/support/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Technical Support page&lt;/a&gt; for additional support resources.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1058399" author="judu" created="Mon, 12 Oct 2015 20:33:29 +0000"  >&lt;p&gt;Ok, so here is the build log of my currently running instances, which are still encountering the same crash stacktrace.&lt;br/&gt;
Built against libboost 1.59.0.&lt;br/&gt;
Note: we built 2.6 ourselves too without a problem. The v8 used is the one provided by the source, not a system one.&lt;/p&gt;</comment>
                            <comment id="1056863" author="judu" created="Fri, 9 Oct 2015 19:14:25 +0000"  >&lt;p&gt;Build log of mongodb.&lt;/p&gt;</comment>
                            <comment id="1055902" author="judu" created="Thu, 8 Oct 2015 20:25:22 +0000"  >&lt;p&gt;Sorry, I&apos;m testing something about the build, so build log is coming. I&apos;m not forgetting!&lt;/p&gt;</comment>
                            <comment id="1050913" author="judu" created="Mon, 5 Oct 2015 12:44:37 +0000"  >&lt;p&gt;Crash log for conn1606 with all the logs. (Same as crash.log, but with all the logs from all the connections in case I erased a bit too much in crash.log.)&lt;/p&gt;</comment>
                            <comment id="1050911" author="judu" created="Mon, 5 Oct 2015 12:43:34 +0000"  >&lt;p&gt;Crash log with all (and only) the &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1606&amp;#93;&lt;/span&gt; in it.&lt;/p&gt;</comment>
                            <comment id="1050871" author="ramon.fernandez" created="Mon, 5 Oct 2015 11:43:51 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judu&quot; class=&quot;user-hover&quot; rel=&quot;judu&quot;&gt;judu&lt;/a&gt;, can you please send a longer part of the log? In particular I&apos;m looking for more details about &lt;tt&gt;conn447&lt;/tt&gt;, which is the one involved in the segfault:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-10-02T18:44:55.505+0000 F -        [conn447] Invalid access at address: 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-10-02015-10-02T18:44:55.521+0000 F -        [conn447] Got signal: 11 (Segmentation fault).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Also, how did you install this &lt;tt&gt;mongod&lt;/tt&gt; instance? Did you use a package manager or did you build it from sources? If the latter, can you please send the command line used to build it?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1050345" author="judu" created="Fri, 2 Oct 2015 22:00:33 +0000"  >&lt;p&gt;Thanks for your quick answer &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Ok, first, we do not use SELinux nor grsecurity.&lt;br/&gt;
I took the example systemd service file from the mongodb doc. With:&lt;br/&gt;
LimitFSIZE=infinity&lt;br/&gt;
LimitCPU=infinity&lt;br/&gt;
LimitAS=infinity&lt;br/&gt;
LimitNOFILE=64000&lt;br/&gt;
LimitNPROC=64000&lt;/p&gt;

&lt;p&gt;So, I&apos;m attaching the result of `ldd mongod` if you can find anything strange in it.&lt;/p&gt;

&lt;p&gt;In the log I attached, there is a mapReduce at the beginning. It&apos;s the last operation of that kind.&lt;/p&gt;

&lt;p&gt;I ran that mapReduce again, and succeeded to crash the primary node by doing so. I included the log starting at the first mapReduce command.&lt;/p&gt;</comment>
                            <comment id="1050187" author="ramon.fernandez" created="Fri, 2 Oct 2015 19:26:18 +0000"  >&lt;p&gt;Sorry you&apos;ve run into this &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judu&quot; class=&quot;user-hover&quot; rel=&quot;judu&quot;&gt;judu&lt;/a&gt;. The crash shows the problem is happening inside the V8 engine:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongod(_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x16) [0x11ca146]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We&apos;ve seen similar issues in V8 when the machine is configured with SELinux, grsecurity, or imposes other limitations that affect V8&apos;s memory management. Can you please elaborate on the configuration for the affected node? Also, can you provide details of what operations this node was running when it crashed? I&apos;m looking for javascript-related operations like using &lt;tt&gt;$where&lt;/tt&gt; or mapReduce.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="92545" name="crash.log" size="31819" author="judu" created="Mon, 5 Oct 2015 12:43:34 +0000"/>
                            <attachment id="92546" name="full-crash.log" size="1708165" author="judu" created="Mon, 5 Oct 2015 12:44:37 +0000"/>
                            <attachment id="92456" name="mapReduce-crash.log" size="21062" author="judu" created="Fri, 2 Oct 2015 21:48:45 +0000"/>
                            <attachment id="92452" name="mongod-ldd.txt" size="1605" author="judu" created="Fri, 2 Oct 2015 21:13:46 +0000"/>
                            <attachment id="93494" name="mongodb-build-server.log" size="2353234" author="judu" created="Fri, 9 Oct 2015 19:14:25 +0000"/>
                            <attachment id="92422" name="primary-crash.log" size="62033" author="judu" created="Fri, 2 Oct 2015 17:52:37 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 2 Oct 2015 19:22:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 2 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 2 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>judu</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrksxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsdgyf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkzz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>