<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:24:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30652] PRIMARY switching over to SECONDARY frequently</title>
                <link>https://jira.mongodb.org/browse/SERVER-30652</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&lt;br/&gt;
We deployed a 3 node replica set (1-PRIMARY, 1-SECONDARY and 1-ARBITER) for POC purpose&lt;br/&gt;
When trying to load around 100K collections to the database, the SECONDARY could not keep with the load and went out of sync and shutdown&lt;br/&gt;
The load continued as there was still a PRIMARY but it then crashed with the below symptoms&lt;/p&gt;

&lt;p&gt;1. Throughout the load, we see errors like &lt;br/&gt;
    a. &lt;span class=&quot;error&quot;&gt;&amp;#91;conn270741&amp;#93;&lt;/span&gt; thread over memory limit, cleaning up, current: 498k&lt;br/&gt;
    b. Socket say send() Broken pipe&lt;br/&gt;
    c. Fri Aug 11 03:08:21.466 I COMMAND  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn165804&amp;#93;&lt;/span&gt; serverStatus was very slow: &lt;/p&gt;
{ after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after locks: 0, after network: 0, after opLatencies: 0, after opcounters: 0, after opcountersRepl: 0, after repl: 6589, after security: 6589, after sharding: 6589, after storageEngine: 6589, after tcmalloc: 6589, after wiredTiger: 6589, at end: 6589 }

&lt;p&gt;2. We see that the PRIMARY  transitioned to SECONDARY multiple times (around 14 times in a day) and an election took place and was transitioned back to PRIMARY&lt;/p&gt;

&lt;p&gt;Fri Aug 11 03:03:32.034 D REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Scheduling heartbeat to xsj-db1:27030 at 2017-08-11T10:03:33.978Z&lt;br/&gt;
Fri Aug 11 03:03:32.041 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Member xsj-db2:27030 is now in state ARBITER&lt;br/&gt;
Fri Aug 11 03:03:32.041 D REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Scheduling heartbeat to xsj-db2:27030 at 2017-08-11T10:03:34.041Z&lt;br/&gt;
Fri Aug 11 03:03:32.042 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;replExecDBWorker-0&amp;#93;&lt;/span&gt; transition to SECONDARY&lt;/p&gt;

&lt;p&gt;Fri Aug 11 03:03:43.143 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; Starting an election, since we&apos;ve seen no PRIMARY in the past 10000ms&lt;/p&gt;

&lt;p&gt;Fri Aug 11 03:03:43.297 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; election succeeded, assuming primary role in term 26&lt;br/&gt;
Fri Aug 11 03:03:43.298 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplicationExecutor&amp;#93;&lt;/span&gt; transition to PRIMARY&lt;/p&gt;

&lt;p&gt;All the while we have checked and found that the ARBITER has been up&lt;/p&gt;

&lt;p&gt;3. After the switchover to secondary for the 14th time, the election does not take place and the number of connections increase to 32k all the while the max number of connections was only around 415. After reaching 32k connections the database is hung and below error is recorded continously until the database process crashes&lt;/p&gt;

&lt;p&gt;Fri Aug 11 22:35:42.361 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;thread1&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable&lt;br/&gt;
Fri Aug 11 22:35:42.365 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;thread1&amp;#93;&lt;/span&gt; failed to create service entry worker thread for 172.19.154.189:9621&lt;/p&gt;

&lt;p&gt;Can you please suggest what should be the action taken during such occurences?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Tanveer&lt;/p&gt;</description>
                <environment></environment>
        <key id="416482">SERVER-30652</key>
            <summary>PRIMARY switching over to SECONDARY frequently</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="tanveermadan@gmail.com">Tanveer Madan Marate</reporter>
                        <labels>
                    </labels>
                <created>Mon, 14 Aug 2017 21:50:01 +0000</created>
                <updated>Fri, 9 Feb 2018 18:51:48 +0000</updated>
                            <resolved>Thu, 18 Jan 2018 22:05:51 +0000</resolved>
                                    <version>3.4.4</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1778399" author="thomas.schubert" created="Thu, 18 Jan 2018 22:05:38 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tanveermadan%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;tanveermadan@gmail.com&quot;&gt;tanveermadan@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Sorry for the delay getting back to you. From our investigation we have not identified a bug in MongoDB contributing to this behavior. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-users group.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1648173" author="tanveermadan@gmail.com" created="Mon, 14 Aug 2017 23:03:30 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;I have updated the log files and also the diagnostic.data directory&lt;br/&gt;
Please let me know if any other files are required&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Tanveer&lt;/p&gt;</comment>
                            <comment id="1648137" author="thomas.schubert" created="Mon, 14 Aug 2017 22:00:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tanveermadan%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;tanveermadan@gmail.com&quot;&gt;tanveermadan@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for reporting this behavior. So we can better understand what is going on here, would you please provide the complete log files and an archive of the diagnostic.data directory in the $dbpath for both the primary and secondary nodes?&lt;/p&gt;

&lt;p&gt;I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/af042e89-cf86-4f65-ad5e-5c9086f8249e.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; for you to use. Files uploaded to this portal are only visible to MongoDB employees investigating this issue and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 14 Aug 2017 22:00:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 3 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 3 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>tanveermadan@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htczrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht4qnz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htcluf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>