<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:27:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31591] HMAC key refresh thread holds onto an OperationContext</title>
                <link>https://jira.mongodb.org/browse/SERVER-31591</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The default key refresh time is 3 months: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/keys_collection_manager.cpp#L37&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/master/src/mongo/db/keys_collection_manager.cpp#L37&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;There&apos;s a background thread responsible for refreshing keys. That thread renews its &lt;tt&gt;OperationContext&lt;/tt&gt; on each iteration. However, an iteration (I believe) can sleep for the full three months when newly negotiated keys are set to expire. Holding this &lt;tt&gt;OperationContext&lt;/tt&gt; open can block WT tables from being dropped. Generally speaking, an &lt;tt&gt;OperationContext&lt;/tt&gt; should be short-lived.&lt;/p&gt;</description>
                <environment></environment>
        <key id="446307">SERVER-31591</key>
            <summary>HMAC key refresh thread holds onto an OperationContext</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="randolph@mongodb.com">Randolph Tan</assignee>
                                    <reporter username="daniel.gottlieb@mongodb.com">Daniel Gottlieb</reporter>
                        <labels>
                    </labels>
                <created>Mon, 16 Oct 2017 19:47:20 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:09 +0000</updated>
                            <resolved>Thu, 2 Nov 2017 19:45:33 +0000</resolved>
                                    <version>3.6.0-rc1</version>
                                    <fixVersion>3.6.0-rc3</fixVersion>
                                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="1715870" author="xgen-internal-githook" created="Thu, 2 Nov 2017 19:45:12 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Randolph Tan&apos;, &apos;username&apos;: &apos;renctan&apos;, &apos;email&apos;: &apos;randolph@10gen.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31591&quot; title=&quot;HMAC key refresh thread holds onto an OperationContext&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31591&quot;&gt;&lt;del&gt;SERVER-31591&lt;/del&gt;&lt;/a&gt; Don&apos;t hold active RecoveryUnit while going to long sleep in HMAC key refresh&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ba06ea5e2c4d5c54d6ae715691a306eb0254d7a4&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ba06ea5e2c4d5c54d6ae715691a306eb0254d7a4&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1711386" author="kaloian.manassiev" created="Fri, 27 Oct 2017 17:48:33 +0000"  >&lt;p&gt;I spoke with Eric more about this offline and now I understand the problems with releasing the session inside of abandonSnaphshot. Because of this I agree that the next best thing we can do is what Randolph proposed - to use two different OperationContexts.&lt;/p&gt;</comment>
                            <comment id="1711334" author="milkie" created="Fri, 27 Oct 2017 17:13:55 +0000"  >&lt;p&gt;I think for this one we should simply switch to using a timed condvar instead of the one provided by OperationContext.&lt;/p&gt;</comment>
                            <comment id="1711330" author="milkie" created="Fri, 27 Oct 2017 17:11:32 +0000"  >&lt;p&gt;I actually tried just returning the session to the cache when txnClose() was called.  Unfortunately, the code is currently architected such that you can hold cursor objects past the end of a transaction, and the cursor needs access to the session object in order to find its way back into the session&apos;s cursor cache.  We&apos;d have to find each and every place that destructs cursors outside of a WriteUnitOfWork (or implicit read transaction), and that sounds like a very lengthy and difficult task.&lt;br/&gt;
Also, if you return a session to the cache without all of its cursors returned to it, invariants fire.  I&apos;m not sure I could even design something that could support having the cursors come back later (or have the cursors inserted into a different session&apos;s cursor cache from their original one).&lt;/p&gt;</comment>
                            <comment id="1711318" author="kaloian.manassiev" created="Fri, 27 Oct 2017 17:03:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; told me that you tried destroying and recreating &lt;tt&gt;_session&lt;/tt&gt; in abandonSnapshot, but hit some bugs. Is it an option that we try to make that work?&lt;/p&gt;</comment>
                            <comment id="1711299" author="milkie" created="Fri, 27 Oct 2017 16:54:10 +0000"  >&lt;p&gt;What you want to have happen is to have the destructor run for the _session member variable of WiredTigerRecoveryUnit, and the only way to do that right now is to destruct the recovery unit itself.&lt;/p&gt;</comment>
                            <comment id="1711298" author="milkie" created="Fri, 27 Oct 2017 16:51:17 +0000"  >&lt;p&gt;abandonSnapshot() only ends the current transaction; it does not return the session to the pool.&lt;/p&gt;</comment>
                            <comment id="1711259" author="kaloian.manassiev" created="Fri, 27 Oct 2017 16:24:26 +0000"  >&lt;p&gt;The RecoveryUnit is what holds the WT session and this should have been &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b06ccf5d5ea42228680e4d0bfee1af6b227c79bc/src/mongo/db/concurrency/d_concurrency.h#L203&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;released&lt;/a&gt; when the last global lock on an OpCtx is released. Therefore I do not understand what the problem is that we need to shorten the scope of an operation context.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, is it possible that there is a bug in abandonSnapshot?&lt;/p&gt;</comment>
                            <comment id="1701743" author="renctan" created="Tue, 17 Oct 2017 19:42:31 +0000"  >&lt;p&gt;I have thought about this more, one relatively easy work around for the key refresh code is to use a separate OperationContext for the sleep. Fortunately, the code that uses the OperationContext is very contained and can be easily scoped:&lt;/p&gt;

&lt;p&gt;one operation context that is scoped to this block:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.5.13/src/mongo/db/keys_collection_manager_sharding.cpp#L237-L256&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.5.13/src/mongo/db/keys_collection_manager_sharding.cpp#L237-L256&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;and the other that is scoped to this block:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.5.13/src/mongo/db/keys_collection_manager_sharding.cpp#L286-L298&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.5.13/src/mongo/db/keys_collection_manager_sharding.cpp#L286-L298&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1700820" author="schwerin" created="Tue, 17 Oct 2017 02:16:26 +0000"  >&lt;p&gt;I discussed this problem with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mira.carey%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;mira.carey@mongodb.com&quot;&gt;mira.carey@mongodb.com&lt;/a&gt; a couple of weeks ago. I think we&apos;re going to need to think about the unit of interruptibility as we tease apart the pieces of &quot;Client&quot;, &quot;OperationContext&quot; and &quot;TransactionSession&quot; into their appropriate homes. It&apos;s a little sticky, especially for these long-lived service threads. I think it&apos;s going to remain gross in 3.6, but have to get better during 3.8.&lt;/p&gt;</comment>
                            <comment id="1700771" author="spencer" created="Mon, 16 Oct 2017 23:23:23 +0000"  >&lt;p&gt;So the functionality to do conditional variable sleeps through an OperationContext is relatively new (see &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25062&quot; title=&quot;Implement interruptible sleep and condition variable waitFor methods on OperationContext&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25062&quot;&gt;&lt;del&gt;SERVER-25062&lt;/del&gt;&lt;/a&gt;) and something we were hoping to move more code towards, not less, to better unify our handling of interruption, deadline expiration, and shutdown.  Is there a way we could change the code for sleeping on an OperationContext to release whatever resources are associated with the OperationContext (presumably the RecoveryUnit?) while it&apos;s asleep?&lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1700357" author="daniel.gottlieb@10gen.com" created="Mon, 16 Oct 2017 21:37:27 +0000"  >&lt;p&gt;Yeah, coding style issues aside, a raw delete is sufficient. The other consideration being a gnarly surprise if some future maintainer tries to access the recovery unit in some other part of the key refreshing work.&lt;/p&gt;</comment>
                            <comment id="1700286" author="renctan" created="Mon, 16 Oct 2017 21:08:16 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; Oh, I was not paying attention. I thought releaseRecoveryUnit destroys it. In this particular case, I believe it should be sufficient to just destroy the recovery unit since it is not going to be used after it wakes up again anyway. Or am I missing something?&lt;/p&gt;</comment>
                            <comment id="1700267" author="daniel.gottlieb@10gen.com" created="Mon, 16 Oct 2017 20:54:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=renctan&quot; class=&quot;user-hover&quot; rel=&quot;renctan&quot;&gt;renctan&lt;/a&gt; It is the recovery unit, but releasing it is not sufficient. The instance must be destroyed/deleted (as opposed to simply being re-attached after the sleep):&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/operation_context.cpp#L384-L386&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/operation_context.cpp#L384-L386&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;More color for the curious:&lt;br/&gt;
The member that must have its destructor run: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.h#L127&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.h#L127&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;More directly:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L388-L391&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L388-L391&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;which calls:&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L328&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/068afad15322f6e768cafdfde76dfa92575a17cf/src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp#L328&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1700243" author="renctan" created="Mon, 16 Oct 2017 20:40:12 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; What is the exact thing that is blocking WT operations? Is it just the RecoveryUnit, or the entire OperationContext? If it&apos;s the former, would calling releaseRecoveryUnit before it goes to sleep help?&lt;/p&gt;</comment>
                            <comment id="1700149" author="daniel.gottlieb@10gen.com" created="Mon, 16 Oct 2017 19:48:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=acm&quot; class=&quot;user-hover&quot; rel=&quot;acm&quot;&gt;acm&lt;/a&gt; There were a few cases of &lt;tt&gt;OperationContexts&lt;/tt&gt; being held open for indefinitely long times that we&apos;re trying to squash out for 3.6. If you don&apos;t think this can be prioritized in time, please let me know!&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="428152">SERVER-31101</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="445337">SERVER-31573</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 16 Oct 2017 19:55:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 14 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_14262" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>End date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 27 Oct 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 14 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hthyrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht9tjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1956">Sharding 2017-11-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_14261" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>Start date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 16 Oct 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hthkvb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>