<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:40:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35663] Replication recovery does not update the logical clock</title>
                <link>https://jira.mongodb.org/browse/SERVER-35663</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;If a node crashes with unapplied oplog entries, when it starts back up it will apply to the end of its oplog through ReplicationRecoveryImpl::recoverFromOplog. This applies the entries by directly &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a7/src/mongo/db/repl/oplog_applier.cpp#L145&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;calling SyncTail::multiApply&lt;/a&gt; (through &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a7/src/mongo/db/repl/replication_recovery.cpp#L377&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;an OplogApplier)&lt;/a&gt;, which does not update the logical clock, unlike &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a7/src/mongo/db/repl/sync_tail.cpp#L945-L946&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;normal secondary application&lt;/a&gt;. Then when starting up its replication coordinator, the node will &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a7/src/mongo/db/repl/replication_coordinator_impl.cpp#L505-L507&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;asynchronously schedule ReplicationCoordinatorImpl::_finishLoadLocalConfig&lt;/a&gt; which updates the logical clock &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a7/src/mongo/db/repl/replication_coordinator_impl.cpp#L603-L610&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;b&gt;after&lt;/b&gt; it updates the replication coordinator&apos;s lastAppliedOpTime&lt;/a&gt; to the opTime of the latest oplog entry.&lt;/p&gt;

&lt;p&gt;If a request is processed during this window in _finishLoadLocalConfig, when the node goes to compute logical time metadata for the response, it can hit &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0a42e212a77bb187abc3c56795fda44abb49c3ea/src/mongo/db/service_entry_point_common.cpp#L430&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this invariant&lt;/a&gt; because the operationTime, which is typically the lastAppliedOpTime, will be greater than the latest time in the logical clock.&lt;/p&gt;

&lt;p&gt;Two ways to fix this would be to have replication recovery update the logical clock when applying the unapplied oplog entries or to update the global timestamp before updating lastAppliedOpTime when finishing loading the local replica set config.&lt;/p&gt;</description>
                <environment></environment>
        <key id="561024">SERVER-35663</key>
            <summary>Replication recovery does not update the logical clock</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="jack.mulrow@mongodb.com">Jack Mulrow</assignee>
                                    <reporter username="jack.mulrow@mongodb.com">Jack Mulrow</reporter>
                        <labels>
                            <label>sharding-wfbf-day</label>
                    </labels>
                <created>Mon, 18 Jun 2018 20:54:17 +0000</created>
                <updated>Sun, 29 Oct 2023 22:30:39 +0000</updated>
                            <resolved>Thu, 17 Jan 2019 23:14:43 +0000</resolved>
                                    <version>4.0.0-rc5</version>
                    <version>4.1.1</version>
                                    <fixVersion>4.0.7</fixVersion>
                    <fixVersion>4.1.8</fixVersion>
                                    <component>Replication</component>
                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2162261" author="xgen-internal-githook" created="Mon, 25 Feb 2019 21:49:39 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;username&apos;: &apos;jsmulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35663&quot; title=&quot;Replication recovery does not update the logical clock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35663&quot;&gt;&lt;del&gt;SERVER-35663&lt;/del&gt;&lt;/a&gt; Always update the logical clock before advancing the last applied opTime&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bfb0aec447f39633694dcc4418f2a5a5a167937e)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/33388105eecf102c9a18138f751ea8a502eb257c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/33388105eecf102c9a18138f751ea8a502eb257c&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2119473" author="xgen-internal-githook" created="Thu, 17 Jan 2019 23:14:03 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;jsmulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;, &apos;name&apos;: &apos;Jack Mulrow&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35663&quot; title=&quot;Replication recovery does not update the logical clock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35663&quot;&gt;&lt;del&gt;SERVER-35663&lt;/del&gt;&lt;/a&gt; Always update the logical clock before advancing the last applied opTime&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/bfb0aec447f39633694dcc4418f2a5a5a167937e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/bfb0aec447f39633694dcc4418f2a5a5a167937e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2086485" author="judah.schvimer" created="Mon, 10 Dec 2018 23:44:12 +0000"  >&lt;blockquote&gt;
&lt;p&gt;convert it to a dassert since the clock will eventually correct itself and this check is mostly for testing&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Are there any causal consistency violations that can occur if there is a bug here? Should we add an invariant where we move the lastApplied forward that we don&apos;t move it ahead of the logical clock, or should we move the logical clock forward in the same place where we move the lastApplied forward rather than one after another everywhere else? Or will that have a significant performance cost?&lt;/p&gt;

&lt;p&gt;Otherwise lgtm&lt;/p&gt;</comment>
                            <comment id="2078826" author="jack.mulrow" created="Mon, 3 Dec 2018 18:35:07 +0000"  >&lt;p&gt;The underlying problem here seems to be the invariant mentioned in the description expects a relationship between operationTime and clusterTime (i.e. operationTime &amp;lt;= clusterTime), which puts a dependency between the ReplicationCoordinator (where operationTime is &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/service_entry_point_common.cpp#L296-L300&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;retrieved for reads&lt;/a&gt;) and the LogicalClock (where &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e313/src/mongo/db/service_entry_point_common.cpp#L344&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;cluster time is retrieved&lt;/a&gt;) that isn&apos;t always enforced.&lt;/p&gt;

&lt;p&gt;E.g. from the description, when we finish loading the replica set config (which happens after replication recovery), the ReplicationCoordinator&apos;s &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/replication_coordinator_impl.cpp#L641&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;last applied opTime is advanced&lt;/a&gt;, then the ReplicationCoordinator mutex is unlocked before &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/replication_coordinator_impl.cpp#L648&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;advancing the LogicalClock&lt;/a&gt;&#160;(through the setGlobalTimestamp method). If a request is processed during this window, when its logical time metadata is computed, it may receive a correct operationTime but a stale clusterTime, which violates the invariant.&lt;/p&gt;

&lt;p&gt;To fix this, I think it&apos;s sufficient to ensure wherever the last applied opTime can advance we always update the LogicalClock first. The main risk I can see with this approach is an inability to enforce this ordering at compile-time, but re-enabling the invariant should allow us to catch errors during our tests.&lt;/p&gt;

&lt;p&gt;My proposed solution:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Re-enable the failing invariant, but convert it to a dassert since the clock will eventually correct itself and this check is mostly for testing&lt;/li&gt;
	&lt;li&gt;Audit each place the ReplicationCoordinator&apos;s last applied opTime can be advanced and ensure the LogicalClock is updated first. The places I&apos;ve currently found are:
	&lt;ol&gt;
		&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/replication_coordinator_impl.cpp#L332-L333&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;When setting the last opTime in initial sync&lt;/a&gt;&lt;/li&gt;
		&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/replication_coordinator_impl.cpp#L639-L648&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;When the replication config is finished loading&lt;/a&gt;&lt;/li&gt;
		&lt;li&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/replication_coordinator_impl.cpp#L3055-L3060&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;When resetting the last opTimes in the ReplicationCoordinator&lt;/a&gt; (e.g. after rollback)&lt;/li&gt;
		&lt;li&gt;During secondary oplog application (the order is already correct here, so I would just update &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/sync_tail.cpp#L889-L890&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this comment&lt;/a&gt;)&lt;/li&gt;
		&lt;li&gt;When an &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e3132/src/mongo/db/repl/oplog.cpp#L420-L422&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;oplog entry is committed&lt;/a&gt;&#160;(this also seems correct, since the time &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b12875f9e313/src/mongo/db/repl/oplog.cpp#L164&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;is reserved in the clock&lt;/a&gt;&#160;before the WriteUnitOfWork is committed)&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=misha.tyulenev&quot; class=&quot;user-hover&quot; rel=&quot;misha.tyulenev&quot;&gt;misha.tyulenev&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, what do you think of the proposed solution?&lt;/p&gt;</comment>
                            <comment id="1963440" author="xgen-internal-githook" created="Wed, 1 Aug 2018 20:44:17 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;, &apos;username&apos;: &apos;jsmulrow&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35663&quot; title=&quot;Replication recovery does not update the logical clock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35663&quot;&gt;&lt;del&gt;SERVER-35663&lt;/del&gt;&lt;/a&gt; Remove clusterTime &amp;gt;= operationTime invariant&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bc67f3ea909bc1c13d46aa439f44cb88cbf67374)&lt;br/&gt;
Branch: v3.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/19523a841554dd16580afa3059acd58bee0ecb9e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/19523a841554dd16580afa3059acd58bee0ecb9e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1924251" author="xgen-internal-githook" created="Mon, 18 Jun 2018 21:44:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;jsmulrow&apos;, &apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35663&quot; title=&quot;Replication recovery does not update the logical clock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35663&quot;&gt;&lt;del&gt;SERVER-35663&lt;/del&gt;&lt;/a&gt; Remove clusterTime &amp;gt;= operationTime invariant&lt;/p&gt;

&lt;p&gt;(cherry picked from commit bc67f3ea909bc1c13d46aa439f44cb88cbf67374)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/a9a9e398d86591c7559ba26580a38c2d439d7835&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/a9a9e398d86591c7559ba26580a38c2d439d7835&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1924228" author="xgen-internal-githook" created="Mon, 18 Jun 2018 21:28:03 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;jsmulrow&apos;, &apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35663&quot; title=&quot;Replication recovery does not update the logical clock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35663&quot;&gt;&lt;del&gt;SERVER-35663&lt;/del&gt;&lt;/a&gt; Remove clusterTime &amp;gt;= operationTime invariant&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/bc67f3ea909bc1c13d46aa439f44cb88cbf67374&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/bc67f3ea909bc1c13d46aa439f44cb88cbf67374&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1162558">SERVER-46257</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1170097">SERVER-46308</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 18 Jun 2018 21:28:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 50 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 50 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>64.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jack.mulrow@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0w8f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htrqhz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2451">Sharding 2018-08-13</customfieldvalue>
    <customfieldvalue id="2481">Sharding 2018-09-10</customfieldvalue>
    <customfieldvalue id="2482">Sharding 2018-09-24</customfieldvalue>
    <customfieldvalue id="2538">Sharding 2018-10-08</customfieldvalue>
    <customfieldvalue id="2539">Sharding 2018-10-22</customfieldvalue>
    <customfieldvalue id="2540">Sharding 2018-11-05</customfieldvalue>
    <customfieldvalue id="2639">Sharding 2018-12-17</customfieldvalue>
    <customfieldvalue id="2640">Sharding 2018-12-31</customfieldvalue>
    <customfieldvalue id="2725">Sharding 2019-01-14</customfieldvalue>
    <customfieldvalue id="2726">Sharding 2019-01-28</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu0ihr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>