<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:53:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39867] Temper ticket count recovery during periods of low replication lag</title>
                <link>https://jira.mongodb.org/browse/SERVER-39867</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The flow control mechanism reduces the number of ticket acquisitions permitted when replication lag reaches a certain threshold. Presently, when replication lag is low, the number of flow control ticket acquisitions permitted is increased too aggressively, leading to oscillations in throughput. Oscillations can be mitigated by tempering the increase in the number of ticket acquisitions when lag is low.&lt;/p&gt;</description>
                <environment></environment>
        <key id="705970">SERVER-39867</key>
            <summary>Temper ticket count recovery during periods of low replication lag</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="daniel.gottlieb@mongodb.com">Daniel Gottlieb</assignee>
                                    <reporter username="maria.vankeulen@mongodb.com">Maria van Keulen</reporter>
                        <labels>
                    </labels>
                <created>Wed, 27 Feb 2019 18:24:05 +0000</created>
                <updated>Sun, 29 Oct 2023 22:23:28 +0000</updated>
                            <resolved>Fri, 3 May 2019 13:41:20 +0000</resolved>
                                                    <fixVersion>4.1.11</fixVersion>
                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2233500" author="xgen-internal-githook" created="Fri, 3 May 2019 13:41:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;daniel.gottlieb@mongodb.com&apos;, &apos;name&apos;: &apos;Daniel Gottlieb&apos;, &apos;username&apos;: &apos;dgottlieb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39867&quot; title=&quot;Temper ticket count recovery during periods of low replication lag&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39867&quot;&gt;&lt;del&gt;SERVER-39867&lt;/del&gt;&lt;/a&gt;: Temper ticket penalties when commit lag is close to the threshold.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/cd3f10254f5b7fd93c2e4065a996dad74d5b1fd2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/cd3f10254f5b7fd93c2e4065a996dad74d5b1fd2&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2229346" author="daniel.gottlieb@10gen.com" created="Tue, 30 Apr 2019 15:47:23 +0000"  >&lt;p&gt;In this comment I refer to a primary and a secondary. This is to simplify the scenario of a primary and two secondaries where the secondaries process writes at equal rates.&lt;/p&gt;

&lt;p&gt;I tried a different algorithm for flow control that reduce oscillations (i.e: more predictable majority latencies, with trade-offs) in scenarios where a secondary can process exactly 1/4 the operations the primary can (a severe degradation). The original algorithm will, when lagged, have a primary accept writes at half the rate as the secondary.&lt;/p&gt;

&lt;p&gt;The problem of the steady-state oscillations in the original algorithm is that when a primary notices it&apos;s lagged, it puts on the breaks very hard; processing only half of what the secondary can. When the majority lag drops under a threshold (5 seconds), the primary will quickly increase its throughput. Even though the secondary empties the queue of writes to replicate, the 5 second &quot;runway&quot; a primary has is enough to aggressively outpace the secondary again before deciding it should slow down.&lt;/p&gt;

&lt;p&gt;The solution to this (while keeping a &quot;threshold&quot; that determines whether to throttle based on the secondary/versus increase the ticket allocation) is to actually let the primary accept writes roughly on pace with secondaries when lag is near the threshold. As the lag gets worse, the primary will scale down the number of writes it will accept.&lt;/p&gt;

&lt;p&gt;In essence, this is accomplished with a function that looks like:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;k ^ ((lag - threshold)/threshold)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;when &lt;tt&gt;lag &amp;gt; threshold&lt;/tt&gt; for some constant &lt;tt&gt;0 &amp;lt; k &amp;lt; 1&lt;/tt&gt;. The following simulations use k = 0.5 and threshold = 5 seconds. Thus at 10 seconds of lag, the primary will reduce writes to half of what a secondary is processing (roughly, I&apos;ve added another 95% factor such that lag=5 seconds has the primary accepting a few less writes than the secondary can do).&lt;/p&gt;

&lt;p&gt;Using &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt; oscillation script that simulates flow control, I ran four different algorithms, two of the old playing with the rate of increase variable (when the primary determines the replica set is &quot;healthy&quot;, a multiplier of 1.1 and 1.05) and two on the new algorithm also playing with the rate of increase variable, for parity. I took two screenshots of FTDC. One showing the whole range. Specifically, this covers the max lag when starting out. The second zooms in on the steady state oscillations to compute the average/max latencies as well as throughputs there.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/215531/215531_maxlag.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/215532/215532_oscillations.png&quot; width=&quot;900&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;What we see in the first screenshot is that max lag is the same on both algorithms. This makes sense because the workloads both have primaries outputting &amp;gt; 2X the secondary rate (i.e: when the primary notices lag 5 seconds in, it has already produced &amp;gt; 5 additional seconds of work for the secondary to process). The new algorithm recovers from this a bit quicker because it throttles writes a lot more (if the lag never exceeded 10 seconds, I would expect the old algorithm to &quot;recover&quot; sooner).&lt;/p&gt;

&lt;p&gt;We can also observe that overall, the new algorithm has better average throughput, but worse average majority latency compared to their counterparts.&lt;/p&gt;

&lt;p&gt;Looking at the second image which zooms in on steady state, we can see that a few things:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The period for the oscillation for the new algorithm is shorter.&lt;/li&gt;
	&lt;li&gt;The maximum majority latency for the new algorithm is smaller.&lt;/li&gt;
	&lt;li&gt;The new algorithm is smoother.&lt;/li&gt;
	&lt;li&gt;The new algorithm has a higher average latency.&lt;/li&gt;
	&lt;li&gt;The new algorithm has a higher steady-state throughput.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Some of those observations are, once again, due to the severe degradation of the secondary which results in primaries getting a big head start to pick up throughput momentum and demonstrate their processing superiority.&lt;/p&gt;

&lt;p&gt;Running the new/old algorithms against &lt;tt&gt;laggy-7&lt;/tt&gt; however did not demonstrate any conclusive differences, theoretical or otherwise.&lt;/p&gt;

&lt;p&gt;The simulation used the following code (the old algorithm is commented out on top, the new algorithm follows with the ternary expanded):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        # pri_rate_throttled = min(pri_rate, int(last_pri_rate * 1.05)) if tlag &amp;lt; throttle_threshold else sec_rate / 2.0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        if tlag &amp;lt; throttle_threshold:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            pri_rate_throttled = min(pri_rate, int(last_pri_rate * 1.05))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        else:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            pri_rate_throttled = 0.95 * (sec_rate * 0.5**((tlag-throttle_threshold)/throttle_threshold))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2215331" author="maria.vankeulen" created="Tue, 16 Apr 2019 17:26:36 +0000"  >&lt;p&gt;This ticket should also make the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e63e9cea147aae4ad2bbad8c6bb393dbc33af0b4/src/mongo/db/storage/flow_control.cpp#L193&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ticket reduction constant&lt;/a&gt; and the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e63e9cea147aae4ad2bbad8c6bb393dbc33af0b4/src/mongo/db/storage/flow_control.cpp#L214&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ticket multiplier and increment constants&lt;/a&gt; user-configurable.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="687534">SERVER-39494</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="215531" name="maxlag.png" size="145270" author="daniel.gottlieb@mongodb.com" created="Tue, 30 Apr 2019 15:06:48 +0000"/>
                            <attachment id="215532" name="oscillations.png" size="135286" author="daniel.gottlieb@mongodb.com" created="Tue, 30 Apr 2019 15:06:53 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 30 Apr 2019 15:47:23 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 40 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-763</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 40 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>maria.vankeulen@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huoyif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr78lr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2908">Storage NYC 2019-05-06</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huokrr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>