<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:08:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-67434] Improve Sync Source Selection with Chained Replication and Flow Control</title>
                <link>https://jira.mongodb.org/browse/SERVER-67434</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi Team,&lt;/p&gt;

&lt;p&gt;Starting in MongoDB v4.2 the &lt;a href=&quot;https://www.mongodb.com/docs/v4.2/replication/#replication-lag-and-flow-control&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Flow Control Mechanism&lt;/a&gt; was introduced in order to limit the rate at which the primary applies its writes with the goal of keeping the majority committed lag under a configurable maximum value of &lt;a href=&quot;https://www.mongodb.com/docs/v4.2/reference/parameters/#param.flowControlTargetLagSeconds&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;flowControlTargetLagSeconds&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;At the same time, and whenever replication chaining is enabled, the sync source of a secondary will be changed if the most recent OpTime of the sync source is more than &lt;em&gt;maxSyncSourceLagSecs&lt;/em&gt; seconds behind another member&apos;s latest oplog entry. This ensures that the sync source is not too far behind other nodes in the set. maxSyncSourceLagSecs is a server parameter and has a default value of 30 seconds.&lt;/p&gt;

&lt;p&gt;The problem is that the value of &lt;em&gt;maxSyncSourceLagSecs&lt;/em&gt; is bigger (3x) than the default value of 10 seconds for &lt;a href=&quot;https://www.mongodb.com/docs/v4.2/reference/parameters/#param.flowControlTargetLagSeconds&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;flowControlTargetLagSeconds&lt;/a&gt; and that can result in primary nodes being throttled by the Flow Control mechanism just because one secondary lags behind while enough secondary nodes to make up a majority also replicate from it. Imagine the following scenario:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Pri (DC1), Sec (DC1), Lagged-Sec (DC2), Chained-Sec (DC3), Chained-Sec (DC3)&lt;/li&gt;
	&lt;li&gt;Sec syncs from Pri, Lagged-Sec syncs from Pri, and Chained-Sec syncs from Lagged-Sec&lt;/li&gt;
	&lt;li&gt;If there are any issues with Lagged-Sec that results in accumulating replication lag between 10 to 30 seconds, it ends up with 3 nodes having lag of above 10 seconds and kicking off Flow Control&lt;/li&gt;
	&lt;li&gt;The above can result in severe impact to applications and it could go on and on if the lag floats between 10 to 30 seconds, or the lag presents as isolated spikes on that very same range.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;If MongoDB were to consider the interplay between &lt;em&gt;maxSyncSourceLagSecs&lt;/em&gt; and &lt;em&gt;flowControlTargetLagSeconds&lt;/em&gt; in enviornments with chained replication enabled and revaluate its sync source before hitting &lt;em&gt;flowControlTargetLagSeconds&lt;/em&gt; (or maybe shortly after?), then situations like the above would be avoided.&lt;/p&gt;

&lt;p&gt;Some options I thought of:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Gossip the value of &lt;em&gt;flowControlTargetLagSeconds&lt;/em&gt; from the current Primary to the other replica set members and automatically adjust &lt;em&gt;maxSyncSourceLagSecs&lt;/em&gt; as a percentage of the former value. This would apply only when replication chaining is enabled.&lt;/li&gt;
	&lt;li&gt;Consider this dynamic adjustment only for nodes that have votes and priority set to 1 or above, meaning that they count towards the majority committed lag/point.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Regards&lt;br/&gt;
Diego&lt;/p&gt;</description>
                <environment></environment>
        <key id="2074126">SERVER-67434</key>
            <summary>Improve Sync Source Selection with Chained Replication and Flow Control</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="1" iconUrl="https://jira.mongodb.org/images/icons/statuses/open.png" description="">Open</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="diego.rodriguez@mongodb.com">Diego Rodriguez</reporter>
                        <labels>
                            <label>former-quick-wins</label>
                            <label>replication</label>
                    </labels>
                <created>Wed, 22 Jun 2022 13:08:52 +0000</created>
                <updated>Wed, 1 Nov 2023 21:46:07 +0000</updated>
                                            <version>5.2.0</version>
                    <version>5.3.1</version>
                    <version>5.2.1</version>
                    <version>5.0.9</version>
                    <version>4.4.15</version>
                    <version>4.2.21</version>
                                                                        <votes>0</votes>
                                    <watches>12</watches>
                                                                                                                <comments>
                            <comment id="4766768" author="diego.rodriguez" created="Mon, 22 Aug 2022 12:31:39 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb@mongodb.com&quot;&gt;daniel.gottlieb@mongodb.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The disadvantage I see with that approach is that we act once the problem is already there: a majority of your nodes are lagging and flow control is already engaged and throttling writes.&lt;/p&gt;

&lt;p&gt;By propagating the flow control configuration you can directly avoid engaging flow control in scenarios like the one above by telling your Secondaries to re-evaluate the sync source if the lag against the source is about to get close to &lt;em&gt;flowControlTargetLagSeconds&lt;/em&gt;.&lt;/p&gt;</comment>
                            <comment id="4642953" author="daniel.gottlieb@10gen.com" created="Mon, 27 Jun 2022 19:18:11 +0000"  >&lt;p&gt;Maybe a simpler alternative than having a primary propagate its flow control configuration is to instead propagate its state, i.e: &quot;I am currently throttling due to flow control&quot;. And using that information to hint to chained secondaries to change their sync source.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001lt4HGQAY, 5006R00001xKA6dQAG]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 27 Jun 2022 19:18:11 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 24 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 24 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>diego.rodriguez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0zjlz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0iefc:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0z5rb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>