<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:47:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-59409] Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary</title>
                <link>https://jira.mongodb.org/browse/SERVER-59409</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, replica set nodes will report it&apos;s configVersion as the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0028db3e9c096e2196e66b1181f5e3c33cc435a3/src/mongo/db/repl/topology_coordinator.cpp#L2152&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;setVersion&lt;/a&gt;&#160;to the RSM for topology management purposes. The TopologyManager tracks the max setVersion it has seen so far, and for any primaries that report a configVersion &amp;lt; maxSetVersion, the RSM will set the status of the node as UNKNOWN because it thinks it&apos;s a&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/9f988d84aaafe4312ae3941a548c2243285a0c15/src/mongo/client/sdam/topology_state_machine.cpp#L286-L288&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stale primary&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;There is an existing race condition where if a user performs a reconfig that bumps the configVersion from V to V + 1 and a new primary is stepped up before it has applied configVersion V + 1, which will cause the replica set to enter a state where the RSM is unable to detect a primary because it thinks the new primary that still reports config version V is stale.&lt;/p&gt;

&lt;p&gt;Consider the following:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Perform reconfig that bumps (configVersion, term) from (V, T) to (V + 1, T). Secondaries are still on (V, T). RSM sets maxSetVersion to V + 1.&lt;/li&gt;
	&lt;li&gt;New primary steps up, sets it&apos;s own configVersion to (V, T + 1).&#160;&lt;/li&gt;
	&lt;li&gt;Old primary will recognize (V, T + 1) as a newer config than (V + 1, T) since term is&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0028db3e9c096e2196e66b1181f5e3c33cc435a3/src/mongo/db/repl/repl_set_config.h#L79&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;given priority when ordering ConfigVersionAndTerm&lt;/a&gt;. So the old primary will fetch the newer config (V, T +1) to replace its own config.&lt;/li&gt;
	&lt;li&gt;RSM will not recognize the new primary as &quot;up-to-date&quot; since it is still reporting setVersion V when maxSetVersion is set to V + 1.&lt;/li&gt;
	&lt;li&gt;RSM will set the new primary status to UNKNOWN, and report a topology of ReplicaSetNoPrimary.&#160;&lt;br/&gt;
 The RSM and replica set stay out of sync with no way to recover without manual intervention.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Since we will&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/0028db3e9c096e2196e66b1181f5e3c33cc435a3/src/mongo/db/repl/repl_set_commands.cpp#L447-L448&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;report that the reconfig failed&lt;/a&gt;&#160;when we fail to replicate the config to the rest of the nodes. This could prompt users to reissue their reconfig on the new primary. However, this can cause failures in our jstests (our stepdown suites in particular). And also, it sounds like it could be problematic that the RSM becomes out of sync with the actual state with the replica set (and is unable to recover) as there are other components that rely on the RSM.&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1853564">SERVER-59409</key>
            <summary>Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="andrew.shuvalov@mongodb.com">Andrew Shuvalov</assignee>
                                    <reporter username="jason.chan@mongodb.com">Jason Chan</reporter>
                        <labels>
                    </labels>
                <created>Tue, 17 Aug 2021 18:28:55 +0000</created>
                <updated>Sun, 29 Oct 2023 21:49:27 +0000</updated>
                            <resolved>Tue, 19 Oct 2021 16:04:54 +0000</resolved>
                                                    <fixVersion>5.3.0</fixVersion>
                    <fixVersion>4.4.11</fixVersion>
                    <fixVersion>5.0.4</fixVersion>
                    <fixVersion>5.1.0-rc1</fixVersion>
                                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="4520280" author="JIRAUSER1256988" created="Mon, 2 May 2022 16:41:36 +0000"  >&lt;p&gt;Even though this is a significant protocol change it doesn&apos;t look to me that this affects the downgrade procedure, the reason:&lt;/p&gt;

&lt;p&gt;This fixes a bug when a race happens between executing a reconfig command and primary of that shard failover. When the race happens the sender of the command gets stuck because the combination of new values is treated as stale and RSM will wait for a refresh forever.&lt;/p&gt;

&lt;p&gt;Downgrade means migrating to the world where this race is possible. So follow the normal downgrade procedure and expect that the race, which is very rare, is now possible. &lt;/p&gt;</comment>
                            <comment id="4187389" author="xgen-internal-githook" created="Sun, 14 Nov 2021 00:05:46 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-61466&quot; title=&quot;Port the RSM { electionId, setVersion } ordering to scanning RSM&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-61466&quot;&gt;&lt;del&gt;SERVER-61466&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10659 Fix race between reconfig replication and stepup&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/18f93bfb7a2931c474aa8ee9dae990a0d5ec8442&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/18f93bfb7a2931c474aa8ee9dae990a0d5ec8442&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4174939" author="xgen-internal-githook" created="Mon, 8 Nov 2021 16:13:11 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; Reverted: Fix race between reconfig replication and stepup&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b701620a0666b27f71468981fa840557d810a2c2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b701620a0666b27f71468981fa840557d810a2c2&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4173401" author="xgen-internal-githook" created="Sun, 7 Nov 2021 01:32:50 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; Fix race between reconfig replication and stepup&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e6bcbca509e5bd789de0184fc422a69845b8dfe7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/e6bcbca509e5bd789de0184fc422a69845b8dfe7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4172983" author="xgen-internal-githook" created="Fri, 5 Nov 2021 23:36:51 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10659 part 3, make the ElectionIdSetVersionPair comparisons tidy&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8e4cec9a8b8dcf41eb5bcfe2824e60c4633667b4&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8e4cec9a8b8dcf41eb5bcfe2824e60c4633667b4&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4172865" author="xgen-internal-githook" created="Fri, 5 Nov 2021 21:50:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10659 part 2 no op change in tests to require term and Set version&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3ad825a4e3ae253c87d3adb25c95627724d03f90&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3ad825a4e3ae253c87d3adb25c95627724d03f90&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4172636" author="xgen-internal-githook" created="Fri, 5 Nov 2021 20:28:16 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10659 part 1, add class ElectionIdSetVersionPair&lt;br/&gt;
Branch: v4.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a7b1936040a8ff2c66d73c8317e6b947a193c70a&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a7b1936040a8ff2c66d73c8317e6b947a193c70a&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4132691" author="xgen-internal-githook" created="Tue, 19 Oct 2021 15:36:01 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10658 Fix race between reconfig replication and stepup parts 3 4&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/d02754325ef96fed4bc926ba22b03479bb5c6b83&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/d02754325ef96fed4bc926ba22b03479bb5c6b83&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4130653" author="xgen-internal-githook" created="Mon, 18 Oct 2021 20:15:42 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10658 Fix race between reconfig replication and stepup part 2&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/39530228b5bbffb13e6c3213d8b2583476b5098e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/39530228b5bbffb13e6c3213d8b2583476b5098e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4129415" author="xgen-internal-githook" created="Mon, 18 Oct 2021 14:25:27 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; BACKPORT-10658 Fix race between reconfig replication and stepup part 1&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/aee30405dfc8be4e69f37eb57b1178f73ec9fc1d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/aee30405dfc8be4e69f37eb57b1178f73ec9fc1d&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4127366" author="xgen-internal-githook" created="Fri, 15 Oct 2021 20:03:00 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; backport BACKPORT-10657: Fix race between reconfig replication and stepup part 4&lt;br/&gt;
Branch: v5.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b996dfbc5135e94b38fb86014c656930cdc2f533&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b996dfbc5135e94b38fb86014c656930cdc2f533&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4126910" author="xgen-internal-githook" created="Fri, 15 Oct 2021 17:26:14 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; backport part 3, make the ElectionIdSetVersionPair comparisons tidy&lt;br/&gt;
Branch: v5.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b69563e8b35f91441bc6b666cba1c09414b27a08&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b69563e8b35f91441bc6b666cba1c09414b27a08&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4126296" author="JIRAUSER1256988" created="Fri, 15 Oct 2021 13:49:33 +0000"  >&lt;p&gt;The Waterfall looks unchanged to me after 9 hours in head and unless there are any objections I&apos;m doing the 5.1 backport now as there are hot BFs attached to that branch, e.g.&#160;BF-21948.&lt;/p&gt;</comment>
                            <comment id="4126262" author="JIRAUSER1256988" created="Fri, 15 Oct 2021 13:31:03 +0000"  >&lt;p&gt;Filed&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1954&quot; title=&quot;SDAM should give priority to electionId over setVersion when updating topology&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1954&quot;&gt;&lt;del&gt;DRIVERS-1954&lt;/del&gt;&lt;/a&gt; for SDAM spec change.&lt;/p&gt;</comment>
                            <comment id="4125452" author="xgen-internal-githook" created="Thu, 14 Oct 2021 23:59:36 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; Fix race between reconfig replication and stepup&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/565f818c1277998b9712fe2c9fc8c6d6158df07b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/565f818c1277998b9712fe2c9fc8c6d6158df07b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4103078" author="xgen-internal-githook" created="Tue, 5 Oct 2021 17:50:41 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; part 3, make the ElectionIdSetVersionPair comparisons tidy&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f6ef384450a3e562a8cb0607b1f50c85c9c47e3c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f6ef384450a3e562a8cb0607b1f50c85c9c47e3c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4097371" author="xgen-internal-githook" created="Fri, 1 Oct 2021 19:11:17 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; part 1, add class ElectionIdSetVersionPair to be used in TopologyStateMachine later&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3a5fdbfcf053b1b165a474705639e0ef0fe28b88&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3a5fdbfcf053b1b165a474705639e0ef0fe28b88&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4095731" author="xgen-internal-githook" created="Fri, 1 Oct 2021 01:39:58 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Andrew Shuvalov&apos;, &apos;email&apos;: &apos;andrew.shuvalov@mongodb.com&apos;, &apos;username&apos;: &apos;shuvalov-mdb&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-59409&quot; title=&quot;Race between reconfig replication and stepup can cause RSM to be stuck in reporting ReplicaSetNoPrimary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-59409&quot;&gt;&lt;del&gt;SERVER-59409&lt;/del&gt;&lt;/a&gt; part 2 no op change in tests to require term and Set version in Hello&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a4d9afb865514b69dfd03ae6867fc416f2b7f708&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a4d9afb865514b69dfd03ae6867fc416f2b7f708&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4060308" author="shane.harvey" created="Tue, 14 Sep 2021 20:23:18 +0000"  >&lt;p&gt;I&apos;m a bit surprised that SDAM was written with the wrong order but I&apos;ll defer to replication to decide on the proper ordering.  &lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;I feel that it should be sufficient to only use the term since we are trying to detect a stale primary&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This section may have the answer:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Replica set members running MongoDB 2.6.10+ or 3.0+ include an integer called &quot;setVersion&quot; and an ObjectId called &quot;electionId&quot; in their hello or legacy hello response. Starting with MongoDB 3.2.0, replica sets can use two different replication protocol versions; electionIds from one protocol version must not be compared to electionIds from a different protocol.&lt;/p&gt;

&lt;p&gt;Because protocol version changes require replica set reconfiguration, clients use the tuple (setVersion, electionId) to detect stale primaries.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#using-setversion-and-electionid-to-detect-stale-primaries&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#using-setversion-and-electionid-to-detect-stale-primaries&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4060203" author="lamont.nelson" created="Tue, 14 Sep 2021 19:51:52 +0000"  >&lt;p&gt;This issue isn&apos;t technically a bug. The RSM is just following the specification. It seems that the tuple ordering implied by the updateRSFromPrimary routine in the spec is actually the opposite of the replication code. The problem is that (setVersion, electionId) is not monotonically increasing, but (electionId, setVersion) is. The RSM is using the first ordering, and this ticket is an example of why it doesn&apos;t work. It makes sense to me that this is the case. The portion of the code (and spec) that is in question is trying to determine if a message is from a stale primary. This is the main purpose of the term in the raft protocol &amp;#8211; similar to the proposal number in paxos. I feel that it should be sufficient to only use the term since we are trying to detect a stale primary, but I might not have considered all the use cases. However, in order to make any changes via repl set reconfig we have to talk to a primary so it makes sense that (electionId, setVersion) would be the ordering. All reconfig changes have to happen under the leadership of some primary since we can only run them from the primary. I&apos;m not sure if (setVersion, electionId) monotonicity was ever true for this reason. Either way, this ticket provides a counter example for the latest version of the code.&lt;/p&gt;</comment>
                            <comment id="4059468" author="shane.harvey" created="Tue, 14 Sep 2021 17:57:37 +0000"  >&lt;p&gt;A spec change starts by creating a DRIVERS ticket followed by a github PR to the specs repo. However, I&apos;m not following what the spec change would be because I thought this issue was a bug in the server itself. The drivers expect (setVersion, electionId) to be monotonically increasing when reported from a primary. Does that expectation no longer hold? What do you propose for the new driver behavior?&lt;/p&gt;</comment>
                            <comment id="4048469" author="JIRAUSER1258790" created="Thu, 9 Sep 2021 20:00:05 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lamont.nelson&quot; class=&quot;user-hover&quot; rel=&quot;lamont.nelson&quot;&gt;lamont.nelson&lt;/a&gt;&#160;Thanks for confirming, I will assign this to Sharding NYC backlog for now.&lt;/p&gt;

&lt;p&gt;One additional thought: Since this will probably change the RSM Spec, will there be a problem for backward compatibility with v3.2.1 and before because &lt;a href=&quot;https://github.com/mongodb/specifications/blob/b07c26dc40d04ac20349f989db531c9845fdd755/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#monotonicity&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;electionId was not guaranteed to be monotonic&lt;/a&gt;&#160;for those versions? Let me know if you need any help from Repl to make that work, thanks.&lt;/p&gt;</comment>
                            <comment id="4048178" author="lamont.nelson" created="Thu, 9 Sep 2021 18:15:47 +0000"  >&lt;p&gt;Since it seems like the changes that need to occur within the server are RSM related we can take this ticket if you like.&lt;/p&gt;</comment>
                            <comment id="4048170" author="lamont.nelson" created="Thu, 9 Sep 2021 18:11:55 +0000"  >&lt;p&gt;This seems like a problem to me. It seems intuitively (to me at least) correct to have the ordering be (term/electionId, version) since term should uniquely identify a primary and the topology changes must occur by talking to the primary. At least within the server we can make the RSM consistent with the repl sub-system.&lt;/p&gt;</comment>
                            <comment id="4048121" author="JIRAUSER1258790" created="Thu, 9 Sep 2021 17:54:12 +0000"  >&lt;p&gt;RSM uses the &lt;tt&gt;(setVersion, electionId)&lt;/tt&gt;&#160;to detect stale primary where setVersion is compared first, but looks like the &lt;tt&gt;electionId&lt;/tt&gt;&#160;is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/171f588acc51178080f0c400bc00a190c67c50e1/src/mongo/db/repl/replication_coordinator_impl.cpp#L4299&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;basically the server&apos;s term.&lt;/a&gt;&#160;However server uses &lt;tt&gt;(term, version)&lt;/tt&gt;&#160;pair where term is compared first, and then the version (same as &lt;tt&gt;setVersion&lt;/tt&gt;). This means that RSM uses an opposite approach than the server to detect stale primary, which seems off. I think this is because there used to be &lt;a href=&quot;https://github.com/mongodb/specifications/blob/b07c26dc40d04ac20349f989db531c9845fdd755/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#monotonicity&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;no guarantee for electionId to be monotonic&lt;/a&gt; before v3.2.1.&lt;/p&gt;

&lt;p&gt;Another interesting finding is that it the Spec has already described &lt;a href=&quot;https://github.com/mongodb/specifications/blob/b07c26dc40d04ac20349f989db531c9845fdd755/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#ignore-setversion-unless-the-server-is-primary&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the same issue here&lt;/a&gt;. However the proposed solution (ignore &lt;tt&gt;setVersion&lt;/tt&gt;&#160;from non-primaries) does not seems to be correct because server A and server B can both report as primary, while reporting &lt;tt&gt;setVersion&lt;/tt&gt;&#160;5 and 4 respectively, so ignoring setVersion from non-primaries does not take effect here to prevent the issue.&#160; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shane.harvey&quot; class=&quot;user-hover&quot; rel=&quot;shane.harvey&quot;&gt;shane.harvey&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lamont.nelson&quot; class=&quot;user-hover&quot; rel=&quot;lamont.nelson&quot;&gt;lamont.nelson&lt;/a&gt;&#160;Could you help confirm if this is a problem with the Spec?&lt;/p&gt;</comment>
                            <comment id="4007016" author="jason.chan" created="Wed, 18 Aug 2021 16:55:02 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shane.harvey&quot; class=&quot;user-hover&quot; rel=&quot;shane.harvey&quot;&gt;shane.harvey&lt;/a&gt;.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;What is the rationale for this order? Drivers&#160;&lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#using-setversion-and-electionid-to-detect-stale-primaries&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;give priority to the setVersion&lt;/a&gt;&#160;not the term.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;We order term first as part of our config consensus protocol so that in scenarios similar to above (when a new node steps up before the reconfig on the old primary is majority committed), the new primary is still considered to have the most &quot;up-to-date config&quot; (with a higher term). If the old primary fails over before replicating configVersionAndTerm (V + 1, T), a new primary can stepUp with config (V, T + 1). When the old primary rejoins the set as a secondary, it knows that (V, T + 1) is a newer config and should fetch it to align with the rest of the cluster.&lt;/p&gt;</comment>
                            <comment id="4007005" author="lamont.nelson" created="Wed, 18 Aug 2021 16:49:22 +0000"  >&lt;p&gt;Would it be possible to not advertise the new setVersion in the hello response until a majority of the nodes have completed the reconfig? &lt;/p&gt;

&lt;p&gt;Also it seems strange that a primary can stepup without having applied the latest reconfig.&lt;/p&gt;</comment>
                            <comment id="4005494" author="shane.harvey" created="Tue, 17 Aug 2021 22:25:38 +0000"  >&lt;p&gt;Yes, I believe this issue would also effect drivers. Drivers expect (setVersion, electionId) to be monotonically increasing. &lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;Old primary will recognize (V, T + 1) as a newer config than (V + 1, T) since term is given priority when ordering ConfigVersionAndTerm. &lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;What is the rationale for this order? Drivers &lt;a href=&quot;https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#using-setversion-and-electionid-to-detect-stale-primaries&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;give priority to the setVersion&lt;/a&gt; not the term. &lt;/p&gt;</comment>
                            <comment id="4005453" author="jason.chan" created="Tue, 17 Aug 2021 21:54:30 +0000"  >&lt;blockquote&gt;&lt;p&gt;Is this an issue for drivers in addition to the RSM?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I think this is possible since IIUC, RSM follows the same SDAM protocol as drivers. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=shane.harvey&quot; class=&quot;user-hover&quot; rel=&quot;shane.harvey&quot;&gt;shane.harvey&lt;/a&gt;, does this sound like an issue for drivers?&lt;/p&gt;</comment>
                            <comment id="4004967" author="judah.schvimer" created="Tue, 17 Aug 2021 19:09:51 +0000"  >&lt;p&gt;Is this an issue for drivers in addition to the RSM?&lt;/p&gt;</comment>
                            <comment id="4004866" author="jason.chan" created="Tue, 17 Aug 2021 18:33:50 +0000"  >&lt;p&gt;Also note: Replication has a history of doing internal reconfigs (like in setFCV), which can cause problems in our stepdown suites&lt;/p&gt;</comment>
                            <comment id="4004857" author="jason.chan" created="Tue, 17 Aug 2021 18:31:06 +0000"  >&lt;p&gt;cc: &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lamont.nelson&quot; class=&quot;user-hover&quot; rel=&quot;lamont.nelson&quot;&gt;lamont.nelson&lt;/a&gt;&#160;to see if the behavior above is a real issue.&lt;/p&gt;

&lt;p&gt;Some potential solutions may include:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Having the RSM take into account the term value when calculating maxSetVersion&lt;/li&gt;
	&lt;li&gt;Have replica set nodes also increment the config version when stepping up&lt;/li&gt;
	&lt;li&gt;Only update maxSetVersion once a majority of replica set nodes report the new version&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1898743">DRIVERS-1954</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1857097">SERVER-59484</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>32.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="22495"><![CDATA[v5.1]]></customfieldvalue>
    <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10012"><![CDATA[Major Change]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Aug 2021 19:09:51 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 40 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Filed &lt;a href=&quot;https://jira.mongodb.org/browse/DRIVERS-1954&quot; title=&quot;SDAM should give priority to electionId over setVersion when updating topology&quot; class=&quot;issue-link&quot; data-issue-key=&quot;DRIVERS-1954&quot;&gt;&lt;strike&gt;DRIVERS-1954&lt;/strike&gt;&lt;/a&gt; to track drivers change</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 40 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>152.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.shuvalov@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>lamont.nelson@mongodb.com</customfieldvalue>
            <customfieldvalue>shane.harvey@mongodb.com</customfieldvalue>
            <customfieldvalue>wenbin.zhu@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzy49z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hypskn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5142">Repl 2021-09-06</customfieldvalue>
    <customfieldvalue id="5143">Repl 2021-09-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16945"><![CDATA[Drivers]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzxqiv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>