<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39096] Prepared transactions and DDL operations can deadlock on a secondary, if a reader blocks on a prepared document</title>
                <link>https://jira.mongodb.org/browse/SERVER-39096</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When a transaction is prepared on secondary, a DBHash transaction on the same collection will have to wait for the prepared transaction to either commit or abort, while holding a global lock in IX mode. If a DDL operation comes in at this moment, the DDL operation will be blocked since it needs the global lock in X mode. There might be commit or abort command coming after the DDL in the oplog, however, they don&apos;t have a chance to apply.&lt;/p&gt;

&lt;p&gt;This isn&apos;t a problem on primary, since a concurrent commit / abort command can come in while DBHash is blocked, however secondaries serialize all operations. This won&apos;t be solved by acquiring weaker locks for DDL operations either, because DBHash may be holding a IX lock on one collection, but waiting for a prepared transaction in another collection. For example, there are two collection C1 and C2. The sequence of operations on a secondary is:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;(Oplog Application) A transaction with a single write in C1 is prepared.&lt;/li&gt;
	&lt;li&gt;Read-only transactional DBHash comes in, holding the collection lock on C1 and C2. DBHash is waiting for the prepared transaction to commit or abort.&lt;/li&gt;
	&lt;li&gt;(Oplog Application) A DDL on C2 is applied, waiting after the DBHash for collection lock.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The prepared transaction and the DDL operation should both have succeeded on primary since they don&apos;t conflict.&lt;/p&gt;

&lt;p&gt;Only snapshot read on secondary causes this problem since they cannot yield. There might be a way to change DBHash to use a special snapshot isolation mechanism instead of read-only transaction. Locking, prepare-waiting and DDL operations (e.g. dropCollection) have to be considered when designing such an approach. In the short term, we&#160;perhaps need to disable read-only transaction on secondaries.&lt;/p&gt;</description>
                <environment></environment>
        <key id="674940">SERVER-39096</key>
            <summary>Prepared transactions and DDL operations can deadlock on a secondary, if a reader blocks on a prepared document</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="2">Won&apos;t Fix</resolution>
                                        <assignee username="xiangyu.yao@mongodb.com">Xiangyu Yao</assignee>
                                    <reporter username="siyuan.zhou@mongodb.com">Siyuan Zhou</reporter>
                        <labels>
                    </labels>
                <created>Fri, 18 Jan 2019 20:34:26 +0000</created>
                <updated>Thu, 18 Apr 2019 21:09:28 +0000</updated>
                            <resolved>Mon, 4 Feb 2019 21:25:28 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="2137351" author="xiangyu.yao" created="Mon, 4 Feb 2019 21:25:16 +0000"  >&lt;p&gt;Discussed offline with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt; and we decided a better solution for the case &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt; described would be to make secondary acquire finer-grained locks (same as the primary). Filed &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39372&quot; title=&quot;Make secondary lock acquisition for DDL operations consistent with behavior on primary&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39372&quot;&gt;&lt;del&gt;SERVER-39372&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Closing this ticket as &apos;Won&apos;t Fix&apos; because we are not going to implement lock yielding during prepare conflict retry.&lt;/p&gt;</comment>
                            <comment id="2136632" author="judah.schvimer" created="Mon, 4 Feb 2019 15:55:07 +0000"  >&lt;p&gt;&lt;tt&gt;createIndexes&lt;/tt&gt; was just an example. I expect &lt;tt&gt;collMod&lt;/tt&gt;, &lt;tt&gt;create&lt;/tt&gt;, &lt;tt&gt;drop&lt;/tt&gt;, &lt;tt&gt;dropIndexes&lt;/tt&gt;, and &lt;tt&gt;renameCollection&lt;/tt&gt; all could. &lt;/p&gt;</comment>
                            <comment id="2136431" author="milkie" created="Mon, 4 Feb 2019 13:40:28 +0000"  >&lt;p&gt;We now have code like this for secondary batch application: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/8cbbac8636b5cfae5dd84f308ca9092b0da8b549/src/mongo/db/repl/sync_tail.cpp#L372&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/8cbbac8636b5cfae5dd84f308ca9092b0da8b549/src/mongo/db/repl/sync_tail.cpp#L372&lt;/a&gt;&lt;br/&gt;
I&apos;m not sure why createIndexes is specially exempt from taking a global lock &amp;#8211; are there still metadata commands other than createIndexes that could cause the deadlock you describe?&lt;/p&gt;</comment>
                            <comment id="2136420" author="judah.schvimer" created="Mon, 4 Feb 2019 13:25:27 +0000"  >&lt;p&gt;To clarify the proposed problem, for (2) this isn&apos;t necessarily a DBHash, this could be any secondary read with an &lt;tt&gt;afterClusterTime&lt;/tt&gt;. &lt;/p&gt;

&lt;p&gt;You make a really good point. The original problem was predicated on the DDL operation not conflicting with the prepared transaction. This was possible because the DBHash blocked on the prepared collection (through a prepare conflict) and the DDL operation&apos;s collection (through a lock conflict) separately. Due to two-phase locking, DBHash held onto the DDL operation&apos;s lock while blocked on the prepared transaction. Without 2PL (i.e. outside of a transaction) are there any secondary reads (like DBHash maybe?) that hold onto one collection lock while reading from another collection at the same time?&lt;/p&gt;

&lt;p&gt;Furthermore, we often take stronger locks on secondaries than we do on primaries. If we have a &lt;tt&gt;createIndex&lt;/tt&gt; operation that takes a collection lock for collection A on the primary but a global lock on the secondary, I feel like this can still happen. &lt;br/&gt;
1) prepare a transaction on collection &quot;B&quot;&lt;br/&gt;
2) an &lt;tt&gt;afterClusterTime&lt;/tt&gt; read could block on the prepared transaction on collection B while holding a collection B IS lock.&lt;br/&gt;
3) &lt;tt&gt;createIndex&lt;/tt&gt; on collection A taking the global X lock would block on the &lt;tt&gt;afterClusterTime&lt;/tt&gt; read&apos;s global IS lock (from the collection B IS lock).&lt;br/&gt;
4) commit never has a chance to run&lt;/p&gt;

&lt;p&gt;This is able to happen on the primary because the &lt;tt&gt;createIndex&lt;/tt&gt; takes a more granular lock. &lt;/p&gt;</comment>
                            <comment id="2135535" author="xiangyu.yao" created="Fri, 1 Feb 2019 21:28:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, I don&apos;t quite understand why this deadlock can still happen with normal afterClusterTime reads on secondary.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;The afterClusterTime read can hold a collection IS lock, and block on a prepare conflict. A DDL op being applied can then block on the afterClusterTime read to release its lock, and the prepared transaction will block on the DDL op to finish being applied for its commit oplog entry to be applied.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;As described, the sequence is like the following:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;prepare a txn on collection &quot;a&quot;&lt;/li&gt;
	&lt;li&gt;(from DBHash) do an afterClusterTime read on collection &quot;a&quot;, get prepare conflict and keep retrying (holding IS lock on &quot;a&quot; forever)&lt;/li&gt;
	&lt;li&gt;do a DDL operation on collection &quot;a&quot; which requires X lock on &quot;a&quot; --&amp;gt; this gets blocked forever&lt;/li&gt;
	&lt;li&gt;commit that txn --&amp;gt; this never has the chance to run&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Imagine we don&apos;t have that afterClusterTime read, then the serialized operations are:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;prepare a txn on collection &quot;a&quot;&lt;/li&gt;
	&lt;li&gt;do a DDL operation on collection &quot;a&quot;&lt;/li&gt;
	&lt;li&gt;commit that txn&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;But such sequence can never be generated by the primary: on primary, Step 1 holds an IX lock on &quot;a&quot; until Step 3; therefore, Step 2 (which requires X lock on &quot;a&quot;) could never be done in between Step 1 and 3.&lt;/p&gt;</comment>
                            <comment id="2126478" author="geert.bosch" created="Thu, 24 Jan 2019 20:53:42 +0000"  >&lt;p&gt;It may not always be safe to yield locks (for example inside &lt;tt&gt;DBDirectClient&lt;/tt&gt;). Maybe safer to throw an exception in this case and wait somewhere in query.&lt;/p&gt;</comment>
                            <comment id="2126443" author="milkie" created="Thu, 24 Jan 2019 20:22:24 +0000"  >&lt;p&gt;Please proceed; I&apos;m fine with that plan.&lt;/p&gt;</comment>
                            <comment id="2126432" author="siyuan.zhou@10gen.com" created="Thu, 24 Jan 2019 20:18:05 +0000"  >&lt;p&gt;Given that secondary transaction is banned in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39139&quot; title=&quot;Remove testing support for secondary transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39139&quot;&gt;&lt;del&gt;SERVER-39139&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, do you mind my changing the title to &quot;Support yielding behavior on prepare conflict retry&quot; and re-assign it to Storage team?&lt;/p&gt;</comment>
                            <comment id="2124547" author="judah.schvimer" created="Wed, 23 Jan 2019 18:43:01 +0000"  >&lt;p&gt;My understanding is the WCE retry happens slightly differently from prepare conflict retrying, but I think yes, assuming both are included I think it would solve the issue by letting the DDL op acquire the lock it needs to complete.&lt;/p&gt;</comment>
                            <comment id="2124408" author="milkie" created="Wed, 23 Jan 2019 17:27:28 +0000"  >&lt;p&gt;I wonder if we should add lock yielding behavior to all Write Conflict Exception retrying, for plan executors that support yielding.&#160; (Not for every WCE retry, but yielding locks according to the yield policy.)&lt;/p&gt;

&lt;p&gt;Would that solve the issue here?&lt;/p&gt;</comment>
                            <comment id="2124341" author="judah.schvimer" created="Wed, 23 Jan 2019 16:54:41 +0000"  >&lt;p&gt;Even removing secondary transactions, per conversation with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tess.avitabile&quot; class=&quot;user-hover&quot; rel=&quot;tess.avitabile&quot;&gt;tess.avitabile&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn&quot;&gt;max.hirschhorn&lt;/a&gt;, we think this can still happen with normal &lt;tt&gt;afterClusterTime&lt;/tt&gt; reads on secondaries. The &lt;tt&gt;afterClusterTime&lt;/tt&gt; read can hold a collection IS lock, and block on a prepare conflict. A DDL op being applied can then block on the &lt;tt&gt;afterCLusterTime&lt;/tt&gt; read to release its lock, and the prepared transaction will block on the DDL op to finish being applied for its commit oplog entry to be applied. One idea for fixing this is to make reads yield their locks when they block on prepare conflicts. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=geert.bosch&quot; class=&quot;user-hover&quot; rel=&quot;geert.bosch&quot;&gt;geert.bosch&lt;/a&gt;, do you see any problem with this?&lt;/p&gt;</comment>
                            <comment id="2122614" author="tess.avitabile" created="Tue, 22 Jan 2019 14:29:50 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=max.hirschhorn&quot; class=&quot;user-hover&quot; rel=&quot;max.hirschhorn&quot;&gt;max.hirschhorn&lt;/a&gt;, I think it is important that we disable secondary transactions, since they are a drain on productivity, both for BF investigation and for design decisions, as &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt; described. It was a great thing for 4.0 that we were able to perform background dbhash checks, but now that secondaries keep transaction state, we cannot support secondary transactions without a holistic design. I propose that we disable secondary transactions and background dbhash checks now and design a solution for background dbhash checks at the end of the Prepare project if we have time.&lt;/p&gt;</comment>
                            <comment id="2121133" author="max.hirschhorn@10gen.com" created="Fri, 18 Jan 2019 23:56:22 +0000"  >&lt;blockquote&gt;
&lt;p&gt;2. Read-only transactional DBHash comes in, holding the collection lock on C1 and C2. DBHash is waiting for the prepared transaction to commit or abort.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=siyuan.zhou&quot; class=&quot;user-hover&quot; rel=&quot;siyuan.zhou&quot;&gt;siyuan.zhou&lt;/a&gt;, why does the dbhash command need to wait for the prepared transaction to commit or abort? I feel like we should be able to either (a) have the &lt;tt&gt;run_check_repl_dbhash_background.js&lt;/tt&gt; hook pick a &lt;tt&gt;clusterTime&lt;/tt&gt; that is earlier than the timestamp associated with any prepared transactions, or (b) change the dbhash command to return &lt;tt&gt;SnapshotUnavailable&lt;/tt&gt; if there&apos;s a prepared transaction on the collection.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="683624">SERVER-39372</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="736582">SERVER-40594</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="741394">SERVER-40723</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="676205">SERVER-39139</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="681289">SERVER-39321</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 18 Jan 2019 23:56:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 1 week, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 1 week, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>siyuan.zhou@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
            <customfieldvalue>xiangyu.yao@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujqd3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8ezz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2761">Storage NYC 2019-02-11</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hujcmf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>