<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:15:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7841] Segmentation faults and lost replicaset seed on rs.remove()</title>
                <link>https://jira.mongodb.org/browse/SERVER-7841</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We were taking an old MongoDB server out of a replicaset (one of two shards), to be replaced with a newer build of the system. The machine to be taken out was called mongo2-1, and it was the primary.&lt;/p&gt;

&lt;p&gt;I first stepped it down from Primary:&lt;br/&gt;
rs.stepDown();&lt;/p&gt;

&lt;p&gt;mongo2-4 was elected the new primary. That went very smoothly, and mongo2-4 was serving requests with little or no effect on the site from the transition.&lt;/p&gt;

&lt;p&gt;I let that run for a minute, and since everything looked fine, I did&lt;br/&gt;
rs.remove(&quot;mongo2-1.prod.trello.local:27017&quot;) on mongo2-4.&lt;/p&gt;

&lt;p&gt;The new primary (mongo2-4) logged some very strange replica set behavior:&lt;br/&gt;
Mon Dec  3 21:56:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn219311&amp;#93;&lt;/span&gt; starting new replica set monitor for replica set rs2 with seed of mongo2-1.prod.trello.local:27017&lt;br/&gt;
Mon Dec  3 21:56:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn219311&amp;#93;&lt;/span&gt; successfully connected to seed mongo2-1.prod.trello.local:27017 for replica set rs2&lt;/p&gt;


&lt;p&gt;Note that the seed there is the replica set member I had just removed.&lt;/p&gt;

&lt;p&gt;Then MongoDB on mongo2-4 seg faulted:&lt;/p&gt;

&lt;p&gt;Mon Dec  3 21:57:00 Invalid access at address: 0xffffffffffffffe0&lt;/p&gt;

&lt;p&gt;Mon Dec  3 21:57:00 Got signal: 11 (Segmentation fault).&lt;/p&gt;

&lt;p&gt;Mon Dec  3 21:57:00 Backtrace:&lt;br/&gt;
0xa90d79 0xa91350 0x7f588eb5acb0 0x5d3f2d 0x5d9d01 0x5da4b0 0xa20db3 0xa2163b 0xa27c45 0xa2837e 0xa23b42 0xa6b06e 0x97cc14 0x97e20f 0x940e25 0x9441b1 0x8869d7 0x88df49 0xaa37d6 0x637497&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa90d79&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa91350&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f588eb5acb0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo17ReplicaSetMonitor16_checkConnectionEPNS_18DBClientConnectionERSsbi+0x11ad) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5d3f2d&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo17ReplicaSetMonitorC1ERKSsRKSt6vectorINS_11HostAndPortESaIS4_EE+0x441) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5d9d01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo17ReplicaSetMonitor3getERKSsRKSt6vectorINS_11HostAndPortESaIS4_EE+0x200) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5da4b0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo5Shard7_rsInitEv+0x133) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa20db3&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo5Shard8_setAddrERKSs+0x13b) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa2163b&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15StaticShardInfo6reloadEv+0xa65) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa27c45&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15StaticShardInfo4findERKSs+0x14e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa2837e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo5Shard5resetERKSs+0x42) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa23b42&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo17SplitChunkCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xbee) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa6b06e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a4) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x97cc14&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x97e20f&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x940e25&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(&lt;em&gt;ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1&lt;/em&gt;+0x11e1) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9441b1&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8869d7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x88df49&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa37d6&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x637497&amp;#93;&lt;/span&gt;&lt;/p&gt;


&lt;p&gt;After it restarted, we were consistenly getting socket exceptions in Mongo clients connecting through mongos, and the mongos logs were showing:&lt;/p&gt;

&lt;p&gt;Mon Dec  3 21:57:36 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn18303&amp;#93;&lt;/span&gt; SyncClusterConnection connecting to &lt;span class=&quot;error&quot;&gt;&amp;#91;mongo-config-1.prod.trello.local:27019&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Dec  3 21:57:36 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn18303&amp;#93;&lt;/span&gt; SyncClusterConnection connecting to &lt;span class=&quot;error&quot;&gt;&amp;#91;mongo-config-2.prod.trello.local:27019&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Dec  3 21:57:36 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn18303&amp;#93;&lt;/span&gt; SyncClusterConnection connecting to &lt;span class=&quot;error&quot;&gt;&amp;#91;mongo-config-3.prod.trello.local:27019&amp;#93;&lt;/span&gt;&lt;br/&gt;
Mon Dec  3 21:57:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn18339&amp;#93;&lt;/span&gt; Assertion: 13129:can&apos;t find shard for: mongo2-5.prod.trello.local:27017&lt;br/&gt;
0x5350c2 0x75caba 0x757fd2 0x7f5297 0x5c3526 0x5c1707 0x5ef3ee 0x76fbd9 0x7b60e7 0x7c8691 0x5e8127 0x7f270a3dbe9a 0x7f27098f6cbd&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5350c2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo15StaticShardInfo4findERKSs+0x3aa) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x75caba&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo5Shard5resetERKSs+0x42) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x757fd2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f5297&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBaseERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5&lt;br/&gt;
_S7_bi+0x16) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5c3526&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo15ShardConnection11_finishInitEv+0x137) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5c1707&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursor5_initEv+0x5be) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5ef3ee&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xc59) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x76fbd9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7b60e7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7c8691&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5e8127&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f270a3dbe9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f27098f6cbd&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Finally, we added mongo2-1 back into the replicaset, restarted all mongos and client processes, and things went back to normal. An hour later, we were able to remove mongo2-1 from the replica set and nothing exploded. All told, about a 15 minute site outage.&lt;/p&gt;</description>
                <environment>Ubuntu 12.04.1 LTS on a hi1.4xlarge AWS instance</environment>
        <key id="58294">SERVER-7841</key>
            <summary>Segmentation faults and lost replicaset seed on rs.remove()</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="randolph@mongodb.com">Randolph Tan</assignee>
                                    <reporter username="brettkiefer">Brett Kiefer</reporter>
                        <labels>
                            <label>crash</label>
                            <label>replicaset</label>
                    </labels>
                <created>Tue, 4 Dec 2012 18:31:37 +0000</created>
                <updated>Fri, 15 Feb 2013 15:06:48 +0000</updated>
                            <resolved>Thu, 6 Dec 2012 17:29:17 +0000</resolved>
                                    <version>2.0.3</version>
                                                    <component>Replication</component>
                    <component>Sharding</component>
                    <component>Stability</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="208027" author="brettkiefer" created="Thu, 6 Dec 2012 14:05:36 +0000"  >&lt;p&gt;Yes, we are scheduled to upgrade to 2.0.8, and I agree that SERVER 5166 and SERVER 5833 (linked from SERVER 5110) look like the same issue, so hopefully we won&apos;t see this post-upgrade. Thank you.&lt;/p&gt;</comment>
                            <comment id="207550" author="renctan" created="Wed, 5 Dec 2012 22:55:35 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;Would it be possible to upgrade to 2.0.8? We made a couple of fixes in mongos replica set handling, most notably, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5110&quot; title=&quot;ReplicaSetMonitor::check not thread safe wrt _master &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5110&quot;&gt;&lt;del&gt;SERVER-5110&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="31652">SERVER-5110</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 5 Dec 2012 16:01:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 10 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 10 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>brettkiefer</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnfov:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrkwkn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>35747</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;The description field is in narrative format with stack traces. The cluster is 2 shards, each with a 3-machine replica set with one node as a 1 hour delay slave, and a total of about 300Gb of actual data.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlflr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>