<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:22:13 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-10085] Heartbeats can time out due to high network latency while fetching oplog batches</title>
                <link>https://jira.mongodb.org/browse/SERVER-10085</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #EEEEEE;border-color: #ccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #ccc;background-color: #6CB33F;&quot;&gt;&lt;b&gt;MongoDB Status as of September 30th, 2013&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #EEEEEE;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;br/&gt;
When a secondary requests the next batch of the oplog from the primary, it holds an internal lock while waiting for the data to come back over the network. This same lock is required to service heartbeat requests.  High latency and other network issues between nodes can cause the next batch of oplog data to take some time to retrieve, resulting in heartbeat requests timing out.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;br/&gt;
This issue can result in repeated and unnecessary replica set failover.  It is present in versions of MongoDB prior to and including v2.4.5.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;SOLUTION&lt;/b&gt;&lt;br/&gt;
The issue has been resolved by not holding the bgsync mutex while waiting for the network.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WORKAROUNDS&lt;/b&gt;&lt;br/&gt;
Improving the latency and reliability of your network will help to alleviate symptoms.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;PATCHES&lt;/b&gt;&lt;br/&gt;
Production release v2.4.6 contains the fix for this issue, and production release v2.6.0 will contain the fix as well.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Detailed description:&lt;/p&gt;

&lt;p&gt;bgsync::produce holds the BackgroundSync::_mutex through the call to r.tailingQueryGTE, which fetches the next batch of data from the primary&apos;s oplog.  If it takes a long time to get a response from the primary then heartbeats may start timing out as heartbeats also require getting the BackgroundSync::_mutex.  The fix is to change bgsync::produce to call r.tailingQueryGTE outside of the _mutex lock.&lt;/p&gt;

&lt;p&gt;&lt;br class=&quot;atl-forced-newline&quot; /&gt;
&lt;br class=&quot;atl-forced-newline&quot; /&gt;&lt;/p&gt;


&lt;p&gt;Initial description by Eric on June 28:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Now that we&apos;ve pretty much fixed all the failing unit tests, we can see that zBigMapReduce is failing on the Linux Debug builder.&lt;br/&gt;
&lt;a href=&quot;http://buildlogs.mongodb.org/Linux%2064-bit%20DEBUG/builds/2264/test/recent%20failures/zbigMapReduce.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://buildlogs.mongodb.org/Linux%2064-bit%20DEBUG/builds/2264/test/recent%20failures/zbigMapReduce.js&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The problem seems to be that late in the m/r phase, the network seems to break such that the primary and the secondary can no longer see each other (send and recv time out), which causes the primary to step down.  I have no idea what would be causing this.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This failure has been visible since &lt;ins&gt;Linux 64-bit DEBUG Build&lt;/ins&gt; #2260 on June 27, but likely was hidden by simpler bugs. The last green Linux 64-bit DEBUG build was #2200 on June 13 (SHA1 86e76e34e88c).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://buildbot.10gen.cc/builders/Linux%2064-bit%20DEBUG?numbuilds=100&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://buildbot.10gen.cc/builders/Linux%2064-bit%20DEBUG?numbuilds=100&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It is also visible in &lt;ins&gt;Linux 64-bit debug dur off&lt;/ins&gt; builds since #2440 on June 29. Last green build on this builder was #2438 (SHA1 babd275f8818)&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://buildbot.10gen.cc/builders/Linux%2064-bit%20debug%20dur%20off?numbuilds=50&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://buildbot.10gen.cc/builders/Linux%2064-bit%20debug%20dur%20off?numbuilds=50&lt;/a&gt;&lt;/p&gt;</description>
                <environment>buildbot: Linux 64-bit DEBUG, Linux 64-bit debug dur off</environment>
        <key id="81108">SERVER-10085</key>
            <summary>Heartbeats can time out due to high network latency while fetching oplog batches</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="milkie@mongodb.com">Eric Milkie</assignee>
                                    <reporter username="matt.kangas">Matt Kangas</reporter>
                        <labels>
                            <label>buildbot</label>
                    </labels>
                <created>Wed, 3 Jul 2013 04:20:26 +0000</created>
                <updated>Mon, 11 Jul 2016 17:39:53 +0000</updated>
                            <resolved>Tue, 9 Jul 2013 11:57:33 +0000</resolved>
                                                    <fixVersion>2.4.6</fixVersion>
                    <fixVersion>2.5.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="381529" author="auto" created="Mon, 15 Jul 2013 19:19:23 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-10085&quot; title=&quot;Heartbeats can time out due to high network latency while fetching oplog batches&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-10085&quot;&gt;&lt;del&gt;SERVER-10085&lt;/del&gt;&lt;/a&gt; do not hold bgsync mutex while waiting for the network&lt;br/&gt;
Branch: v2.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/cec25e5e779eee081bf2598c19ba1e754a6726c3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/cec25e5e779eee081bf2598c19ba1e754a6726c3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="377010" author="auto" created="Tue, 9 Jul 2013 11:56:30 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;milkie&apos;, u&apos;name&apos;: u&apos;Eric Milkie&apos;, u&apos;email&apos;: u&apos;milkie@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-10085&quot; title=&quot;Heartbeats can time out due to high network latency while fetching oplog batches&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-10085&quot;&gt;&lt;del&gt;SERVER-10085&lt;/del&gt;&lt;/a&gt; do not hold bgsync mutex while waiting for the network&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/6bc6dc7360b9b346f3910cd16fb08d97d32c70b3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/6bc6dc7360b9b346f3910cd16fb08d97d32c70b3&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 3 Jul 2013 15:48:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 31 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 31 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>matt.kangas</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmofr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrrgyv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>74134</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10166" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Tests Written</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10153"><![CDATA[Unneeded]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hri7pb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>