<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:27:36 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-12099] getaddrinfo(hostname:port) - Temporary failure in name resolution</title>
                <link>https://jira.mongodb.org/browse/SERVER-12099</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We&apos;re running a couple of MongoDB clusters in AWS EC2 and the other day one of our clusters started to misbehave.&lt;/p&gt;

&lt;p&gt;It turned out that the PRIMARY node was unable to contact the other nodes in the cluster. In it&apos;s logs we could see the following lines repeated (for brewity I&apos;ve only included one host in the lines, but it showed the same for the third node as well):&lt;/p&gt;

&lt;p&gt;Sat Dec 14 01:55:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; getaddrinfo(&quot;host2&quot;) failed: Temporary failure in name resolution&lt;br/&gt;
Sat Dec 14 01:55:54 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; couldn&apos;t connect to host2:27017: couldn&apos;t connect to server host2:27017&lt;/p&gt;

&lt;p&gt;On the other nodes we observed the following in their logs:&lt;/p&gt;

&lt;p&gt;Sat Dec 14 01:56:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSyncNotifier&amp;#93;&lt;/span&gt; replset tracking exception: exception: 10278 dbclient error communicating with server: host1:27017&lt;br/&gt;
Sat Dec 14 01:56:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet db exception in producer: 10278 dbclient error communicating with server: host1:27017&lt;br/&gt;
Sat Dec 14 01:56:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member host1:27017 is now in state SECONDARY&lt;br/&gt;
Sat Dec 14 01:56:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; not electing self, host1:27017 would veto&lt;/p&gt;

&lt;p&gt;The first thing we tested was to manually add the other nodes (host2 and host3) IP addresses to host1&apos;s /etc/hosts file, and within a minute the issue seemed resolved (i.e. it could talk to the other nodes again), but it was hardly a permanent solution.&lt;/p&gt;

&lt;p&gt;We continued by running rs.stepDown() through the mongo console, removed the manually added lines in /etc/hosts and then restarted mongod on host1. When it booted up it was once again able to connect to the other nodes.&lt;/p&gt;

&lt;p&gt;It should be noted that while host1 was unable to contact either host2 and host3, we were able to use dig to verify that our internal DNS was able to resolve the DNS name, and we could SSH from on host to the other. Our other services, that are using the same DNS server, showed no similar behaviour (i.e. problems with DNS resolving), and we also have an alarm check in place for DNS resolving, which didn&apos;t trigger.&lt;/p&gt;

&lt;p&gt;Another thing of interest is that host1 has been running mongod since 27th of November 2012 while the other hosts (host2 and host3) were recently restarted (23th of November 2013) when we had a similar issue (though then we ended up with two PRIMARY nodes).&lt;/p&gt;

&lt;p&gt;Let me know if there is anything else that you regarding this (i.e. log files, etc, etc) and I&apos;ll see what I can do.&lt;/p&gt;</description>
                <environment>Linux 2.6.35.14-103.47.amzn1.x86_64, Amazon EC2</environment>
        <key id="102218">SERVER-12099</key>
            <summary>getaddrinfo(hostname:port) - Temporary failure in name resolution</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="mthssdrbrg">Mathias S&#246;derberg</reporter>
                        <labels>
                            <label>cluster</label>
                            <label>dns</label>
                            <label>replication</label>
                            <label>replset</label>
                    </labels>
                <created>Sun, 15 Dec 2013 11:27:17 +0000</created>
                <updated>Wed, 10 Dec 2014 23:05:55 +0000</updated>
                            <resolved>Wed, 5 Mar 2014 21:07:14 +0000</resolved>
                                    <version>2.2.2</version>
                                                    <component>Replication</component>
                    <component>Stability</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="470961" author="eliot" created="Sun, 15 Dec 2013 23:01:16 +0000"  >&lt;p&gt;Another cause of this is running out of file descriptors.&lt;br/&gt;
Can you also check that possibility?&lt;/p&gt;</comment>
                            <comment id="470918" author="mthssdrbrg" created="Sun, 15 Dec 2013 12:00:10 +0000"  >&lt;p&gt;Digging deeper, we seem to have found a possible explanation:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://stackoverflow.com/questions/125466/using-glibc-why-does-my-gethostbyname-fail-after-i-dhcp-has-changed-the-dns-ser&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://stackoverflow.com/questions/125466/using-glibc-why-does-my-gethostbyname-fail-after-i-dhcp-has-changed-the-dns-ser&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We set up a test where we migrated from one DNS server to another while mongo was running, and were able to reproduce the exact same behaviour.&lt;/p&gt;

&lt;p&gt;As the above link indicates, it turns out glibc caches the content of /etc/resolv.conf, which as far as I can tell means mongod has to be restarted (on Linux) in order to pick up changes to any DNS server configuration.&lt;/p&gt;

&lt;p&gt;For a long-running server process this seems like a unnecessary limitation, and for glibc the problem can apparently be avoided by explicitly calling res_init (possibly after repeated name resolution failures).&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="55594">DOCS-5700</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 15 Dec 2013 23:01:16 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 9 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 9 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>mthssdrbrg</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrm5fb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrurxb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>93511</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsp1xb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>