<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:00:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-42385] election timeout callbacks are scheduled too frequently</title>
                <link>https://jira.mongodb.org/browse/SERVER-42385</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In the process of investigating a BF, I discovered this behavior:&lt;br/&gt;
&lt;a href=&quot;https://logkeeper.mongodb.org/lobster/build/0b38b5af3083b9d185100bc1ce94b3a5/test/5d379645f84ae81c0108232f#bookmarks=0%2C8967&amp;amp;l=1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://logkeeper.mongodb.org/lobster/build/0b38b5af3083b9d185100bc1ce94b3a5/test/5d379645f84ae81c0108232f#bookmarks=0%2C8967&amp;amp;l=1&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you jump to line 5707 in that log, you can see that &quot;Scheduling election timeout callback at &quot; is happening on average every 2-3 milliseconds, for many seconds.   Additionally, even within one thread, the callback is not being repeatedly scheduled for times exclusively in the future of the previous scheduling; sometimes it goes backwards in time.  This doesn&apos;t seem correct nor efficient.  &lt;br/&gt;
At least, can we change this log message to log level 5 so that it doesn&apos;t affect performance of the test by swamping the log system?&lt;/p&gt;</description>
                <environment></environment>
        <key id="874485">SERVER-42385</key>
            <summary>election timeout callbacks are scheduled too frequently</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="milkie@mongodb.com">Eric Milkie</reporter>
                        <labels>
                    </labels>
                <created>Wed, 24 Jul 2019 18:28:52 +0000</created>
                <updated>Sun, 29 Oct 2023 22:18:48 +0000</updated>
                            <resolved>Thu, 24 Oct 2019 19:13:45 +0000</resolved>
                                                    <fixVersion>4.3.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="2497495" author="xgen-internal-githook" created="Wed, 23 Oct 2019 14:49:01 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Russotto&apos;, &apos;username&apos;: &apos;mtrussotto&apos;, &apos;email&apos;: &apos;matthew.russotto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-42385&quot; title=&quot;election timeout callbacks are scheduled too frequently&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-42385&quot;&gt;&lt;del&gt;SERVER-42385&lt;/del&gt;&lt;/a&gt; Reduce log spam associated with election timeout resets.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/2070d8ca09576b858c61677bc366a25e7ecfecad&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/2070d8ca09576b858c61677bc366a25e7ecfecad&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2414462" author="matthew.russotto" created="Tue, 10 Sep 2019 14:17:52 +0000"  >&lt;p&gt;Eric: Each time we reschedule the timeout, we reschedule it for now() + the election timeout + some random jitter.  The random jitter is independent of the last scheduled timeout, so if we&apos;re rescheduling quickly, the timeout can go backwards.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/96a1a78772038fa29b94dbe92b51b9e9957d3f6f/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L853&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/96a1a78772038fa29b94dbe92b51b9e9957d3f6f/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L853&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;Judah: We currently don&apos;t know if we&apos;re scheduling the election timeout after it actually fired or just pushing it forward, but we could do so (I think it may actually be a bug that we don&apos;t) and then bump the reschedule case up to LOG_FOR_ELECTION(5).  We probably also want to use LOG_FOR_ELECTION(5) instead of LOG(4) here&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/96a1a78772038fa29b94dbe92b51b9e9957d3f6f/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L831&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/96a1a78772038fa29b94dbe92b51b9e9957d3f6f/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp#L831&lt;/a&gt;&lt;/p&gt;
</comment>
                            <comment id="2406977" author="milkie" created="Wed, 4 Sep 2019 13:26:35 +0000"  >&lt;p&gt;Do we have an answer for why the rescheduling isn&apos;t monotonically increasing?  &lt;/p&gt;</comment>
                            <comment id="2405284" author="judah.schvimer" created="Tue, 3 Sep 2019 13:50:46 +0000"  >&lt;p&gt;Should we improve the logging to make it clear this is rescheduling a timeout, and not just scheduling a new one? Or is that always the case and not actually helpful?&lt;/p&gt;</comment>
                            <comment id="2402490" author="matthew.russotto" created="Fri, 30 Aug 2019 20:51:08 +0000"  >&lt;p&gt;This actually occurs because we&apos;re doing a bulk write, and each batch causes us to reschedule the election timeout.  So reducing the logging is probably the right answer.&lt;/p&gt;</comment>
                            <comment id="2402474" author="matthew.russotto" created="Fri, 30 Aug 2019 20:39:41 +0000"  >&lt;p&gt;Oops, I got fooled, there&apos;s actually a wrapper function called ReplicationCoordinatorImpl::_scheduleWorkAt which should prevent this from happening.  &lt;/p&gt;</comment>
                            <comment id="2400563" author="matthew.russotto" created="Thu, 29 Aug 2019 21:23:24 +0000"  >&lt;p&gt;Also the backwards-in-time is because we add jitter to the election timeout; it&apos;s normal.&lt;/p&gt;</comment>
                            <comment id="2400516" author="matthew.russotto" created="Thu, 29 Aug 2019 20:58:11 +0000"  >&lt;p&gt;For heartbeats, we check to see whether the timeout was canceled before attempting to reschedule it.  For elections, we do not.  We unconditionally cancel and reschedule the election timeout.  This means we can get into a loop&lt;/p&gt;

&lt;p&gt;T1: Start election timeout&lt;br/&gt;
T2: Cancel election timeout, start election timeout&lt;br/&gt;
T1: Receive cancel from T2, cancel election timeout, start election timeout&lt;br/&gt;
T2: Receive cancel from T1, cancel election timeout, start election timeout&lt;/p&gt;

&lt;p&gt;(because we use thread pools, it actually isn&apos;t necessary that T1 and T2 are different threads)&lt;/p&gt;

&lt;p&gt;If the callback is canceled, we should not reschedule it in the callback function, nor execute an election.&lt;/p&gt;

&lt;p&gt;It appears a similar bug also exists in priority takeover and catchup takeover, where canceling the takeover starts an election; we should fix this and add testing to make sure it doesn&apos;t happen.&lt;/p&gt;

&lt;p&gt;Somewhat mystifying is the lack of logs indicating that the election started or failed to start.  It is not clear if this is a real thing or a log artifact.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="911804">SERVER-43089</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 29 Aug 2019 20:58:11 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 16 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 16 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvhbhb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hv5lnj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3200">Repl 2019-08-26</customfieldvalue>
    <customfieldvalue id="3201">Repl 2019-09-09</customfieldvalue>
    <customfieldvalue id="3202">Repl 2019-09-23</customfieldvalue>
    <customfieldvalue id="3260">Repl 2019-10-07</customfieldvalue>
    <customfieldvalue id="3261">Repl 2019-10-21</customfieldvalue>
    <customfieldvalue id="3303">Repl 2019-11-04</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hvgxqn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>