<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:14:44 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7507] Random mongos failure to contact whole cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-7507</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;During routine operation of our mongo cluster, the mongos process on one of our app servers became unresponsive (confirmed by ssh&apos;ing to the app server, running mongo, and running &apos;show dbs&apos;).&lt;/p&gt;

&lt;p&gt;Attached is the mongos.log file from when the issue started, until after mongos was manually restarted and recovered. The machine maintained full network connectivity during this time, and DNS names were resolving in shell.&lt;/p&gt;

&lt;p&gt;During this time, the other app server and background worker show clean mongos.logs (just acquiring and unlocking the distributed lock).&lt;/p&gt;

&lt;p&gt;How can we prevent this happening in future? This kind of failure is critical for us, and I&apos;m happy to help debug/diagnose it further.&lt;/p&gt;</description>
                <environment>AWS, Ubuntu 12.04.1 LTS&lt;br/&gt;
2x shards (each shard consists of 2x replicas and 1x abriter)&lt;br/&gt;
&lt;br/&gt;
2x app servers (each running mongos)&lt;br/&gt;
1x background worker (running mongos)</environment>
        <key id="54722">SERVER-7507</key>
            <summary>Random mongos failure to contact whole cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="randolph@mongodb.com">Randolph Tan</assignee>
                                    <reporter username="noizwaves">noizwaves</reporter>
                        <labels>
                            <label>nh-240</label>
                    </labels>
                <created>Tue, 30 Oct 2012 03:13:52 +0000</created>
                <updated>Wed, 10 Dec 2014 23:19:28 +0000</updated>
                            <resolved>Tue, 28 May 2013 14:28:08 +0000</resolved>
                                    <version>2.2.0</version>
                                                    <component>Networking</component>
                    <component>Replication</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="310723" author="barrie" created="Thu, 11 Apr 2013 00:34:46 +0000"  >&lt;p&gt;Adam,&lt;/p&gt;

&lt;p&gt;Are you still seeing this issue? Have you been able to try upgrading to 2.2.4?&lt;/p&gt;

&lt;p&gt;Barrie &lt;/p&gt;</comment>
                            <comment id="264165" author="renctan" created="Tue, 12 Feb 2013 16:15:53 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;Would you mind elaborating on what kind of failure are you seeing? Are you referring to the socket exceptions in the mongos logs?&lt;/p&gt;</comment>
                            <comment id="240912" author="noizwaves" created="Tue, 15 Jan 2013 23:19:40 +0000"  >&lt;p&gt;Hi, have there been any developments with this? I hate to nag but this is causing is sporadic and random critical errors in our system affecting our uptime.&lt;/p&gt;

&lt;p&gt;We are happy to help debug this in any way we can. &lt;/p&gt;</comment>
                            <comment id="219174" author="noizwaves" created="Wed, 19 Dec 2012 12:05:37 +0000"  >&lt;p&gt;Hey, we are consistently seeing these errors every day now. Is there anything more we can do escalate this issue? Happy to debug anything from our end.&lt;/p&gt;

&lt;p&gt;Cheers, Adam&lt;/p&gt;</comment>
                            <comment id="213702" author="noizwaves" created="Thu, 13 Dec 2012 03:50:08 +0000"  >&lt;p&gt;Thanks for the tips Eliot. We&apos;ve updated to 2.2.1 and this did not resolve the issue. We&apos;ve been encountering it more frequently lately, so I&apos;ll try to capture a dump. (We&apos;ve also bumped logging up to vvvvv for the moment as well).&lt;/p&gt;</comment>
                            <comment id="180827" author="eliot" created="Wed, 31 Oct 2012 06:42:15 +0000"  >&lt;p&gt;A little hard to diagnose with this info.&lt;br/&gt;
Few things:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;can you upgrade to 2.2.1 - various fixes could account, though not 100%&lt;/li&gt;
	&lt;li&gt;if it happens again, can you attach with gdb and get a dump so we can look through it?&lt;/li&gt;
	&lt;li&gt;also if it happens again, can you run top and tell us if cpu is spiked or idle?&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="180415" author="noizwaves" created="Tue, 30 Oct 2012 04:47:39 +0000"  >&lt;p&gt;Hi, the issue has happened again to the same machine. This time, mongos was able to come back online. Any guidance on diagnosing this issue would be appreciated.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Adam&lt;/p&gt;</comment>
                            <comment id="180414" author="noizwaves" created="Tue, 30 Oct 2012 04:46:35 +0000"  >&lt;p&gt;mongos log file from second issue occurrence&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="22555" name="mongo_send_error.tar.gz" size="6091253" author="noizwaves" created="Sun, 16 Dec 2012 22:21:44 +0000"/>
                            <attachment id="21242" name="mongos-2.log" size="36860" author="noizwaves" created="Tue, 30 Oct 2012 04:46:35 +0000"/>
                            <attachment id="21240" name="mongos.log" size="62415" author="noizwaves" created="Tue, 30 Oct 2012 03:13:52 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 31 Oct 2012 06:42:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 45 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 45 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>barrie</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>noizwaves</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnjhb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrkafb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>32113</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0ntz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>