<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:55:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40498] Writing transaction oplog entries must not take locks while holding an oplog slot</title>
                <link>https://jira.mongodb.org/browse/SERVER-40498</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;It is illegal to acquire new locks while holding open an oplog slot, since it can cause the deadlock described in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35367&quot; title=&quot;Hold locks in fewer callers of waitForAllEarlierOplogWritesToBeVisible()&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35367&quot;&gt;&lt;del&gt;SERVER-35367&lt;/del&gt;&lt;/a&gt;. We do this in a few places when writing transaction oplog entries:&lt;/p&gt;

&lt;p&gt;When preparing a transaction, we reserve oplog slots &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1022&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1026&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then stash the locker &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1383&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1398&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then take a global lock &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1389&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; or &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1407&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;When aborting a prepared transaction, we reserve an oplog slot &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1413&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then release locks&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1419&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then take a global lock &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1186&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;When committing a prepared transaction, we reserve an oplog slot &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1235&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then release locks &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/transaction_participant.cpp#L1255&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, then take a global lock &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1186&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;(Note that the global lock acquisition for writing oplog entries for unprepared transaction commit is not a problem, since the locker is already holding the global lock.)&lt;/p&gt;
</description>
                <environment></environment>
        <key id="731336">SERVER-40498</key>
            <summary>Writing transaction oplog entries must not take locks while holding an oplog slot</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="siyuan.zhou@mongodb.com">Siyuan Zhou</assignee>
                                    <reporter username="tess.avitabile@mongodb.com">Tess Avitabile</reporter>
                        <labels>
                    </labels>
                <created>Fri, 5 Apr 2019 17:30:23 +0000</created>
                <updated>Sun, 29 Oct 2023 22:22:16 +0000</updated>
                            <resolved>Thu, 9 May 2019 02:22:39 +0000</resolved>
                                                    <fixVersion>4.1.11</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="2239460" author="xgen-internal-githook" created="Thu, 9 May 2019 02:21:58 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;siyuan.zhou@mongodb.com&apos;, &apos;name&apos;: &apos;Siyuan Zhou&apos;, &apos;username&apos;: &apos;visualzhou&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; Writing transaction oplog entries must not take locks while holding an oplog slot.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fa878904816b984e00d980336755f762f91919c3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fa878904816b984e00d980336755f762f91919c3&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2234487" author="xgen-internal-githook" created="Sat, 4 May 2019 00:32:23 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;siyuan.zhou@mongodb.com&apos;, &apos;name&apos;: &apos;Siyuan Zhou&apos;, &apos;username&apos;: &apos;visualzhou&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40498&quot; title=&quot;Writing transaction oplog entries must not take locks while holding an oplog slot&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40498&quot;&gt;&lt;del&gt;SERVER-40498&lt;/del&gt;&lt;/a&gt; Side transaction keeps the same locker&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4fb71c5a1c79b745ef56d53a8264ef5fdd202dda&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4fb71c5a1c79b745ef56d53a8264ef5fdd202dda&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2234476" author="siyuan.zhou@10gen.com" created="Sat, 4 May 2019 00:00:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt;, you&apos;re right. Never mind, I was fooled by the two-phase locking. The final patch is in code review now.&lt;/p&gt;</comment>
                            <comment id="2230479" author="tess.avitabile" created="Wed, 1 May 2019 14:13:17 +0000"  >&lt;blockquote&gt;
&lt;p&gt;I tried a similar approach to acquire a global lock before reserving the oplog slot as in your patch, but even if we do so, the lock will be released before calling OpObserver.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I don&apos;t understand why the lock will be released before calling OpObserver. If the lock is still in scope, it should not be released when the WUOW is committed. The only locks that get released due to two-phase locking when the WUOW is committed are the locks that have gone out of scope.&lt;/p&gt;

&lt;p&gt;I would also be interested to talk about &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40791&quot; title=&quot;Chunk migration clone blocks behind prepared transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40791&quot;&gt;&lt;del&gt;SERVER-40791&lt;/del&gt;&lt;/a&gt; to get a better understanding of that problem.&lt;/p&gt;</comment>
                            <comment id="2230122" author="siyuan.zhou@10gen.com" created="Wed, 1 May 2019 01:40:45 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, I managed to use the same lock in side transaction and to fix for preparing a transaction. However, I found another issue with commitPreparedTransaction. As found in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39926&quot; title=&quot;Migrations may not transfer writes from prepared transactions committed during cloning phase&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39926&quot;&gt;&lt;del&gt;SERVER-39926&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/df3f76da935a45302d7b24b0abe618af6dd34744/src/mongo/db/transaction_participant.cpp#L1159&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;the transaction is committed to storage and its locks are dropped&lt;/a&gt; &lt;em&gt;before&lt;/em&gt; the transaction commit &lt;a href=&quot;https://github.com/mongodb/mongo/blob/df3f76da935a45302d7b24b0abe618af6dd34744/src/mongo/db/transaction_participant.cpp#L1165-L1166&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;op observer runs with a new WUOW&lt;/a&gt;. I tried a similar approach to acquire a global lock before reserving the oplog slot as in your patch, but even if we do so, the lock will be released before calling OpObserver.&lt;/p&gt;

&lt;p&gt;I can think of two ways to solve this. The solutions are also considered in&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40791&quot; title=&quot;Chunk migration clone blocks behind prepared transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40791&quot;&gt;&lt;del&gt;SERVER-40791&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;We could move onPreparedTransactionCommit() to before committing the storage transaction. Since we don&apos;t allow storage transaction commit to fail, both data WT transaction and oplog WT transaction will commit. To make sure the oplog write isn&apos;t visible before the data change so that afterCluterTime read are respected, we need to reserve an OplogSlot without using it and release it after both data and oplog writes commit as we will do in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40870&quot; title=&quot;OpObserverImpl::onTransactionPrepare() reserves OplogSlots rather than using the one reserved by its caller&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40870&quot;&gt;&lt;del&gt;SERVER-40870&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Alternatively, we could release the recovery unit and the locker from the WUOW and manage them manually on prepared commit, so that we only commit the recovery unit in _commitStorageTransaction() instead of committing the WUOW and have the locker outlive both data and oplog writes. We&apos;ll need storage team to weigh in on this solution.&lt;/p&gt;</comment>
                            <comment id="2219473" author="siyuan.zhou@10gen.com" created="Fri, 19 Apr 2019 18:25:55 +0000"  >&lt;p&gt;Assigning this to me as Tess is on vacation next week.&lt;/p&gt;</comment>
                            <comment id="2205160" author="tess.avitabile" created="Mon, 8 Apr 2019 15:02:46 +0000"  >&lt;p&gt;Before we reserve the oplog slot, we will manually make a Locker and take a global IX lock. We will pass this Locker into onTransactionPrepare(). In onTransactionPrepare(), we will pass the Locker to SideTransactionBlock, which will use it to populate the opCtx after it has stashed the current locker.&lt;/p&gt;</comment>
                            <comment id="2204026" author="tess.avitabile" created="Fri, 5 Apr 2019 21:23:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, I am having trouble fixing the first case, for preparing a transaction. The prepare thread is already holding a global lock. However, we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/op_observer_impl.cpp#L1383&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;stash the transaction resources&lt;/a&gt;, so the OperationContext&apos;s current locker is not holding a global lock. The thread must not take a global lock because it is holding an oplog slot. But it is required to have a global lock when writing to the oplog through this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a05ef93e5a6c691f4b25800a2f3d94e273ab5b9f/src/mongo/db/catalog/collection_impl.cpp#L352&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;dassert&lt;/a&gt;. I&apos;m thinking about a few different solutions:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Remove the dassert. That seems like a bad idea.&lt;/li&gt;
	&lt;li&gt;Stash the transaction resources without stashing the locker. I don&apos;t think that will work, since when we commit the WUOW for the oplog write, we will incorrectly release our locks.&lt;/li&gt;
	&lt;li&gt;Expose stashed lockers through the OperationContext somehow. When writing to the oplog, we can check that the OperationContext&apos;s current locker or any stashed locker has a global lock.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Do you have any suggestions?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="732520">SERVER-40522</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="732520">SERVER-40522</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="553274">SERVER-35367</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="585626">SERVER-36534</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 11 Apr 2019 05:02:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 40 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 40 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>30.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hutai7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|huitif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2918">Repl 2019-04-22</customfieldvalue>
    <customfieldvalue id="2919">Repl 2019-05-06</customfieldvalue>
    <customfieldvalue id="2920">Repl 2019-05-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huswrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>