<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:02:48 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-43296] 3.6.13 ReplicaSet freeze during initial sync / batch sync</title>
                <link>https://jira.mongodb.org/browse/SERVER-43296</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We are migrating a 3.2 standalone server to a 3.6.13 sharded/replicated cluster&lt;/p&gt;

&lt;p&gt;We have 5 RHEL7 nodes, lot of RAM, SSD disks :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;node1 : mongos, config_server1&lt;/li&gt;
	&lt;li&gt;node2: mongod_shard1_primary, mongodb_shard2_arbiter&lt;/li&gt;
	&lt;li&gt;node3: mongod_shard1_secondary, config_server2&lt;/li&gt;
	&lt;li&gt;node4:&#160;mongod_shard2_primary, config_server3&lt;/li&gt;
	&lt;li&gt;node5:&#160;mongod_shard2_secondary, mongodb_shard1_arbiter&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;shard secondaries are hidden, priority 0&lt;/p&gt;

&lt;p&gt;write_concerns=1 from clients&lt;/p&gt;

&lt;p&gt;First 10 days after startup, the data ingestion was ok and our dataset reached 100G on each data_shard (we process a live flow + a migration flow from the 3.2 standalone)&lt;/p&gt;

&lt;p&gt;Then for some reason, we had a first crash on both&#160; shard1 primary/secondary&lt;/p&gt;

&lt;p&gt;After this crash the secondary whas some hours behind the primary&lt;/p&gt;

&lt;p&gt;We cannot now stabilize the shard1 replicaset. When we start the shard1 nodes, the r/w performances are very affected and both shard1 primary and secondary end with freezing and deadlock on clients.&lt;/p&gt;

&lt;p&gt;We can see this in log of secondary:&lt;/p&gt;

&lt;p&gt;&lt;em&gt;2019-09-12T11:25:37.926-0400 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replication-4&amp;#93;&lt;/span&gt; Error returned from oplog query (no more query restarts left): NetworkInterfaceExceededTimeLimit: error in fetcher batch callback: Operation timed out&lt;/em&gt;&lt;br/&gt;
&lt;em&gt;2019-09-12T11:25:37.926-0400 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;replication-4&amp;#93;&lt;/span&gt; Finished fetching oplog during initial sync: NetworkInterfaceExceededTimeLimit: error in fetcher batch callback: Operation timed out. Last fetched optime and hash: { ts: Timestamp(1568301901, 325), t: 43 }&lt;span class=&quot;error&quot;&gt;&amp;#91;8532403056184220739&amp;#93;&lt;/span&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If I disable the replication (standalone shard1) it lives as a charm&lt;/p&gt;

&lt;p&gt;If I try to initial sync the secondary, it ends with a freeze after some Gb of data sync&lt;/p&gt;

&lt;p&gt;Actually the shard1 rs seems Ok after a replica sync from direct data file transfer (the network throughput from node2 to node1 to transfer file was ~100M/s)&lt;/p&gt;

&lt;p&gt;But I&apos;m afraid of a crash in case of any secondary/primary lag&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="924473">SERVER-43296</key>
            <summary>3.6.13 ReplicaSet freeze during initial sync / batch sync</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="franck.lefebure@gmail.com">FRANCK LEFEBURE</reporter>
                        <labels>
                    </labels>
                <created>Thu, 12 Sep 2019 16:32:29 +0000</created>
                <updated>Mon, 16 Sep 2019 15:43:08 +0000</updated>
                            <resolved>Mon, 16 Sep 2019 15:43:08 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2421600" author="daniel.hatcher" created="Mon, 16 Sep 2019 15:43:00 +0000"  >&lt;p&gt;Glad to hear it!&lt;/p&gt;</comment>
                            <comment id="2419929" author="franck.lefebure@gmail.com" created="Fri, 13 Sep 2019 18:01:32 +0000"  >&lt;p&gt;Hi Daniel,&lt;/p&gt;

&lt;p&gt;I really appreciate your quick comment.&lt;/p&gt;

&lt;p&gt;Situation has been fixed by, on all data shards :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;enableMajorityReadConcern: false&lt;/li&gt;
	&lt;li&gt;cacheSizeGB: 1 =&amp;gt; default 50% host (we use a third party packaged mongodb, with this damned 1G default configuration.. )&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Theses pages helped me :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40488&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-40488&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://docs.mongodb.com/manual/reference/read-concern-majority/#disable-read-concern-majority&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/reference/read-concern-majority/#disable-read-concern-majority&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2418619" author="daniel.hatcher" created="Thu, 12 Sep 2019 20:06:51 +0000"  >&lt;p&gt;It is possible that this is an environmental issue instead of a bug and we will be limited in the amount of help that we can provide. However, if you provide some files to our &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/c6d836c4-1eed-4e44-b4cf-9e03874bab42.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Secure Uploader&lt;/a&gt; I can take a look. Please rest assured that any files provided to that link will only be viewable by MongoDB engineers.&lt;/p&gt;

&lt;p&gt;I would like the log files and &quot;diagnostic.data&quot; directories from each of the following:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;all shard nodes that had problems&lt;/li&gt;
	&lt;li&gt;the config server Primary&lt;/li&gt;
	&lt;li&gt;the &lt;tt&gt;mongos&lt;/tt&gt;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 12 Sep 2019 20:06:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 21 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 21 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>franck.lefebure@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvptcf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvehfj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvpflr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>