<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:40:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35738] Mongodb bad performance causes heartbeat lose</title>
                <link>https://jira.mongodb.org/browse/SERVER-35738</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Secondary became primary through election when secondary can not find primary.&lt;/p&gt;

&lt;p&gt; Before this time, many time-consuming operations happened at the primary node, which can be seen in the attach file.&lt;/p&gt;

&lt;p&gt;primary 10.0.208.149&#65292;important log&#65306;&lt;br/&gt;
2018-06-17T04:06:44.004	 end connection 10.0.189.100:42238 (2829 connections now open)&lt;br/&gt;
2018-06-17T04:06:45.770	end connection 10.0.189.100:36520 (2827 connections now open)&lt;br/&gt;
2018-06-17T04:06:51.82	&lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Error in heartbeat request to 10.0.189.100:27017; ExceededTimeLimit: Operation timed out&lt;/p&gt;


&lt;p&gt;2018-06-17T04:06:58.242	connection accepted from 10.0.189.100:39064 #206971&lt;br/&gt;
2018-06-17T04:06:58.242	&lt;span class=&quot;error&quot;&gt;&amp;#91;NetworkInterfaceASIO-Replication-0&amp;#93;&lt;/span&gt; Successfully connected to 10.0.189.100:27017&lt;/p&gt;

&lt;p&gt;2018-06-17T04:06:59.888	&lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.189.100:27017 is now in state SECONDARY&lt;br/&gt;
2018-06-17T04:06:59.890	&lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.0.189.100:39066 #2069721 (2831 connections now open)&lt;br/&gt;
2018-06-17T04:06:59.890	end connection 10.0.189.100:39065 (2830 connections now open)&lt;/p&gt;




&lt;p&gt;secondry   10.0.189.100&#65292;important log&lt;/p&gt;

&lt;p&gt;2018-06-17T04:06:41.460	&lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; could not find member to sync from&lt;/p&gt;



&lt;p&gt;2018-06-17T04:06:58.239+0000	&lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Starting an election, since we&apos;ve seen no PRIMARY in the past 10000ms&lt;br/&gt;
2018-06-17T04:06:58.240+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; conducting a dry run election to see if we could be elected	&lt;/p&gt;


&lt;p&gt;2018-06-17T04:06:58.243+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; VoteRequester: Got no vote from 10.0.208.149:27017 because: candidate&apos;s data is staler than mine, resp:&lt;/p&gt;
{ term: 95, voteGranted: false, reason: &quot;candidate&apos;s data is staler than mine&quot;, ok: 1.0 }
&lt;p&gt;	&lt;/p&gt;</description>
                <environment></environment>
        <key id="562827">SERVER-35738</key>
            <summary>Mongodb bad performance causes heartbeat lose</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="nick.brewer">Nick Brewer</assignee>
                                    <reporter username="cuizihang">Zihang Cui</reporter>
                        <labels>
                    </labels>
                <created>Fri, 22 Jun 2018 08:11:17 +0000</created>
                <updated>Tue, 4 Sep 2018 03:02:59 +0000</updated>
                            <resolved>Wed, 1 Aug 2018 23:06:21 +0000</resolved>
                                    <version>3.2.8</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1963645" author="cuizihang" created="Thu, 2 Aug 2018 03:25:57 +0000"  >&lt;p&gt;I check current and available connections in the mongo shell via:&lt;/p&gt;

&lt;p&gt;db.serverStatus().connections&lt;/p&gt;

&lt;p&gt;/* 1 */ { &quot;current&quot; : 3664, &quot;available&quot; : 47536, &quot;totalCreated&quot; : NumberLong(2378823) }&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="1963560" author="nick.brewer" created="Wed, 1 Aug 2018 23:06:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cuizihang&quot; class=&quot;user-hover&quot; rel=&quot;cuizihang&quot;&gt;cuizihang&lt;/a&gt; It&apos;s difficult to compare the diagnostic data against the logs you&apos;ve provided, as the dates are much older (from when this ticket was first opened). That said, I am seeing multiple entries similar to this in the primary logs:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;SocketException handling request, closing client connection: &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;9001&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; socket exception &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;There are a few potential causes for this message, but I would start by looking at your operating system limits to see if they are configured according to our &lt;a href=&quot;http://docs.mongodb.org/manual/administration/production-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;production notes&lt;/a&gt;. It is possible that your &lt;a href=&quot;http://docs.mongodb.org/manual/reference/ulimit/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ulimit settings&lt;/a&gt; for files are too low.&lt;/p&gt;

&lt;p&gt;From your responses, I do not see anything to indicate a bug in the MongoDB server. For MongoDB-related support discussion please post on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-user group.&lt;/p&gt;

&lt;p&gt;-Nick&lt;/p&gt;</comment>
                            <comment id="1957067" author="cuizihang" created="Thu, 26 Jul 2018 04:43:57 +0000"  >&lt;p&gt;Would you please archive (tar or zip) the $dbpath/diagnostic.data directory from each node in the replica set, and attach them to this ticket?&lt;/p&gt;


&lt;p&gt;--------------------------------------------------------------------------------------------&lt;br/&gt;
I have uploaded the three files mentioned above.  primary.zip secondary.zip backup.zip...&lt;/p&gt;

&lt;p&gt;I hope some new discoveries.&lt;/p&gt;

&lt;p&gt;Thank you.&lt;/p&gt;</comment>
                            <comment id="1957062" author="cuizihang" created="Thu, 26 Jul 2018 04:35:31 +0000"  >&lt;p&gt;Secondary became primary again.&lt;/p&gt;

&lt;p&gt; Secondary log:&lt;br/&gt;
2018-07-26T04:01:00.023+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Error in heartbeat request to 10.0.189.100:27017; ExceededTimeLimit: Operation timed out&lt;br/&gt;
2018-07-26T04:01:04.254+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; stepping down from primary, because a new term has begun: 98&lt;br/&gt;
2018-07-26T04:01:04.322+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-1&amp;#93;&lt;/span&gt; transition to SECONDARY&lt;br/&gt;
2018-07-26T04:01:05.413+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member 10.0.189.100:27017 is now in state PRIMARY&lt;/p&gt;

</comment>
                            <comment id="1931152" author="nick.brewer" created="Mon, 25 Jun 2018 21:23:47 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cuizihang&quot; class=&quot;user-hover&quot; rel=&quot;cuizihang&quot;&gt;cuizihang&lt;/a&gt; Sure - we&apos;ll await your findings. &lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Nick&lt;/p&gt;</comment>
                            <comment id="1930035" author="cuizihang" created="Mon, 25 Jun 2018 02:31:51 +0000"  >
&lt;p&gt;$&amp;gt; ls /data/db_dir/mongodb/diagnostic.data/ -lrth&lt;br/&gt;
total 105M&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 20 17:57 metrics.2018-06-20T04-33-26Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 21 06:30 metrics.2018-06-20T17-57-02Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 21 17:01 metrics.2018-06-21T06-30-18Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 22 03:47 metrics.2018-06-21T17-01-49Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 22 14:32 metrics.2018-06-22T03-47-12Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 23 01:27 metrics.2018-06-22T14-32-26Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 23 12:13 metrics.2018-06-23T01-27-54Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 23 23:09 metrics.2018-06-23T12-13-41Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 24 10:10 metrics.2018-06-23T23-09-26Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb  10M Jun 24 21:05 metrics.2018-06-24T10-10-00Z-00000&lt;br/&gt;
&lt;del&gt;rw-r&lt;/del&gt;&lt;del&gt;r&lt;/del&gt;- 1 mongodb mongodb 4.8M Jun 25 02:20 metrics.2018-06-24T21-05-35Z-00000&lt;/p&gt;

&lt;p&gt;Sorry&#65292;file  had been delete by mongo daemon at Jun 17. We will attach them to this ticket when accident happen again. Would you please keep this jira &#8220;WAITING FOR USER&#8221;?&lt;/p&gt;

&lt;p&gt;Thank you.&lt;/p&gt;</comment>
                            <comment id="1928666" author="nick.brewer" created="Fri, 22 Jun 2018 14:05:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=cuizihang&quot; class=&quot;user-hover&quot; rel=&quot;cuizihang&quot;&gt;cuizihang&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Would you please archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory from each node in the replica set, and attach them to this ticket?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Nick&lt;/p&gt;</comment>
                            <comment id="1928455" author="cuizihang" created="Fri, 22 Jun 2018 08:18:00 +0000"  >&lt;p&gt;&quot;Mongodb bad performance causes heartbeat lose&quot; in the title of this issue is our guess to cause election, but we can&apos;t be sure.&lt;/p&gt;


</comment>
                    </comments>
                    <attachments>
                            <attachment id="190107" name="2018-06-17T04-primary.log" size="3427928" author="cuizihang" created="Fri, 22 Jun 2018 07:56:35 +0000"/>
                            <attachment id="190105" name="2018-06-17T04-secondary-with-priority-0.log" size="42427" author="cuizihang" created="Fri, 22 Jun 2018 07:56:53 +0000"/>
                            <attachment id="190106" name="2018-06-17T04-secondary.log" size="3256781" author="cuizihang" created="Fri, 22 Jun 2018 07:56:47 +0000"/>
                            <attachment id="192679" name="backup.zip" size="6755908" author="cuizihang" created="Thu, 26 Jul 2018 04:39:12 +0000"/>
                            <attachment id="192680" name="primary.zip" size="3596041" author="cuizihang" created="Thu, 26 Jul 2018 04:39:11 +0000"/>
                            <attachment id="192681" name="secondary.zip" size="9155809" author="cuizihang" created="Thu, 26 Jul 2018 04:39:22 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 22 Jun 2018 14:05:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 28 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 28 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>nick.brewer</customfieldvalue>
            <customfieldvalue>cuizihang</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu17av:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hts0c7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;we don&apos;t konw&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0tk7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>