<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:44:20 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36877] Primary slows down when a secondary becomes down</title>
                <link>https://jira.mongodb.org/browse/SERVER-36877</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello!&lt;/p&gt;

&lt;p&gt;We have the following replica set: 1 arbiter, 1 primary and 1 secondary.&lt;/p&gt;

&lt;p&gt;versions:&lt;/p&gt;

&lt;p&gt;primary: mongodb-org-server 3.6.5&lt;/p&gt;

&lt;p&gt;secondary: mongodb-org-server 3.6.6&lt;/p&gt;

&lt;p&gt;arbiter:&#160;mongodb-org-server 3.6.5&lt;/p&gt;

&lt;p&gt;If the secondary goes down (stop, for example) and becomes &quot;not reachable/healthy&quot;, after some period of time (from 1 to 2 hours) the primary will slow down several times: CPU usage on the primary server will increase significantly, amount of avalible concurrent transactions will decrease and become unstable.&lt;/p&gt;

&lt;p&gt;If the secondary becomes up the primary will return to normal state almost immediatly. Or if we exclude the secondary from replica set, primary will also return to normal state and will show ususal performance.&lt;br/&gt;
It is easy to reproduce - just to stop secondary and to wait performance degradation for short perioud of time.&lt;/p&gt;

&lt;p&gt;Usually we have from 1000 to 30000 updates and from 1000 to 6000 reads per second. We also use only primarypreferred read preference and C# driver to connect to DB.&lt;/p&gt;

&lt;p&gt;What is wrong? How can we fix or avoid such unexpected behavior?&lt;/p&gt;

&lt;p&gt;Thanks in advance!&lt;/p&gt;</description>
                <environment></environment>
        <key id="595010">SERVER-36877</key>
            <summary>Primary slows down when a secondary becomes down</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="nick.brewer">Nick Brewer</assignee>
                                    <reporter username="ahrenizm">Maxim Noskov</reporter>
                        <labels>
                    </labels>
                <created>Mon, 27 Aug 2018 07:34:10 +0000</created>
                <updated>Mon, 4 Jul 2022 18:43:58 +0000</updated>
                            <resolved>Wed, 29 Aug 2018 15:26:22 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="4655942" author="JIRAUSER1269677" created="Mon, 4 Jul 2022 18:43:58 +0000"  >&lt;p&gt;Nick,&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;When your replica set is not able to satisfy the read concern, your primary is required to store oplog data in the cache&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Any reason why primary has to store the oplog data in cache? Isn&apos;t oplog data already present on disk? Why not use that?&lt;/p&gt;</comment>
                            <comment id="1988848" author="nick.brewer" created="Wed, 29 Aug 2018 15:26:12 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahrenizm&quot; class=&quot;user-hover&quot; rel=&quot;ahrenizm&quot;&gt;ahrenizm&lt;/a&gt; Glad to hear it worked. &lt;/p&gt;

&lt;p&gt;-Nick&lt;/p&gt;</comment>
                            <comment id="1988498" author="ahrenizm" created="Wed, 29 Aug 2018 06:33:18 +0000"  >&lt;p&gt;Thank you!&lt;/p&gt;

&lt;p&gt;replication.enableMajorityReadConcern: false&#160;helped solve the problem.&lt;/p&gt;

&lt;p&gt;In the future, we will think about&#160;replacing the arbiter with a new data-bearing node&lt;/p&gt;</comment>
                            <comment id="1987619" author="nick.brewer" created="Tue, 28 Aug 2018 15:05:02 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahrenizm&quot; class=&quot;user-hover&quot; rel=&quot;ahrenizm&quot;&gt;ahrenizm&lt;/a&gt; A few things I want to note about disabling the &quot;majority&quot; read concern:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Newer features such as transactions (in 4.0 and above) and change streams require the &quot;majority&quot; read concern.&lt;/li&gt;
	&lt;li&gt;Replacing the arbiter with a new data-bearing node is preferable to utilizing this option.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;With regard to your question:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;if I replace arbitr with a secondary, what about split-brain? Or will the server select a new primary based on the priority?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;A voting majority is necessary to select a primary; it cannot be accomplished solely via priority. The safest process would be to retainthe existing arbiter, &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/expand-replica-set/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;add a new data-bearing member&lt;/a&gt;, and then remove the arbiter once you&apos;ve confirmed that the new member is added.&lt;/p&gt;

&lt;p&gt;-Nick&lt;/p&gt;</comment>
                            <comment id="1987015" author="nick.brewer" created="Mon, 27 Aug 2018 20:35:17 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahrenizm&quot; class=&quot;user-hover&quot; rel=&quot;ahrenizm&quot;&gt;ahrenizm&lt;/a&gt; Another option: with your current PSA setup, you should be able to prevent this behavior by disabling read concern majority in your mongod configuration file(s):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;replication:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    enableMajorityReadConcern: false&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;-Nick&lt;/p&gt;
</comment>
                            <comment id="1986676" author="ahrenizm" created="Mon, 27 Aug 2018 17:10:39 +0000"  >&lt;p&gt;The answer to your question:&lt;br/&gt;
yes, when i remove the secondary from the replica set, i am keeping the arbiter as a member of it.&lt;br/&gt;
About your second suggest: if I replace arbitr with a secondary, what about split-brain? Or will the server select a new primary based on the priority?&lt;/p&gt;</comment>
                            <comment id="1986413" author="nick.brewer" created="Mon, 27 Aug 2018 14:50:08 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahrenizm&quot; class=&quot;user-hover&quot; rel=&quot;ahrenizm&quot;&gt;ahrenizm&lt;/a&gt; MongoDB 3.6 enables the &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/3.6/#read-concern&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;majority&lt;/tt&gt;&lt;/a&gt; read concern automatically. This majority is calculated against the total number of nodes in your replica set (3), however, it can only be serviced by nodes that contain data (2, in your case). With your secondary down, you have only a single data bearing node, so your replica set is not able to satisfy the read concern. &lt;/p&gt;

&lt;p&gt;When your replica set is not able to satisfy the read concern, your primary is required to store oplog data in the cache, which can result in the sort of resource utilization you&apos;re seeing. I would suggest:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Attempt to avoid extended downtime on the secondary&lt;/li&gt;
	&lt;li&gt;Replace your arbiter with an additional data-bearing secondary, which will allow you to fulfill the read concern even with one secondary offline&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;One question: when you remove the secondary from the replica set, are you keeping the arbiter as a member of it? &lt;/p&gt;

&lt;p&gt;&lt;b&gt;Edit:&lt;/b&gt; To be able to satisfy a possible incoming query with read concern majority, the primary is forced to store all updates that occurred since the last write acknowledged by a majority of data-bearing nodes (written to both the primary and secondary, in this case) in the cache. &lt;/p&gt;

&lt;p&gt;Under typical operation, the majority &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/replSetGetStatus/#replSetGetStatus.optimes.lastCommittedOpTime&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;commit point&lt;/a&gt; moves along as new data is inserted, and older versions of the data may be successfully evicted. However, in this case, the secondary went down, which meant the majority commit point could not progress. This resulted performance degradation due to  increased cache pressure as the primary continued to take writes without being able to evict older versions of the data. &lt;/p&gt;

&lt;p&gt;-Nick&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 27 Aug 2018 14:50:08 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 31 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>tejas.jadhav@gojek.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 31 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>ahrenizm</customfieldvalue>
            <customfieldvalue>nick.brewer</customfieldvalue>
            <customfieldvalue>tejas.jadhav@gojek.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu6euf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htwy4v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu613r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>