<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:46:17 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18014] Dropping a collection can block creating a new collection for an extended time under WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-18014</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #EEEEEE;border-color: #ccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #ccc;background-color: #6CB33F;&quot;&gt;&lt;b&gt;Issue Status as of Apr 29, 2015&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #EEEEEE;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;br/&gt;
Because of an interaction between the process for dropping and creating collections with the WiredTiger storage engine, the process of dropping a collection and freeing resources used by that collection can block creating new collections.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;br/&gt;
Workloads that involve large volumes of collection drop operations may experience performance degredation.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;WORKAROUNDS&lt;/b&gt;&lt;br/&gt;
None.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;AFFECTED VERSIONS&lt;/b&gt;&lt;br/&gt;
3.0.0, 3.0.1, 3.0.2&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FIX VERSION&lt;/b&gt;&lt;br/&gt;
The fix is included in the 3.0.3 production release.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;h6&gt;&lt;a name=&quot;Originaldescription&quot;&gt;&lt;/a&gt;Original description&lt;/h6&gt;

&lt;p&gt;Dropping a collection can take a long time under WiredTiger because can require freeing a large number of allocated buffers. See &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17907&quot; title=&quot;B-tree eviction blocks access to collection for extended period under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17907&quot;&gt;&lt;del&gt;SERVER-17907&lt;/del&gt;&lt;/a&gt; for information on reproducing.&lt;/p&gt;

&lt;p&gt;While this is occuring creating a new collection may be blocked for a substantial time. The create was accomplished in this test by inserting a record into a non-existent collection, so the stack traces below show the createCollection happening within insertOne. It is blocked in two different places:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;From A to B it&apos;s blocked waiting for the db lock. This occurs synchronously with the drop command itself, which is busy freeing buffers.&lt;/li&gt;
	&lt;li&gt;From B to C it&apos;s blocked waiting to update the metadata table. This occurs while synchronously with dropAllQueued (not shown in the screenshot below), which is also busy freeing buffers.&lt;br/&gt;
Total timeline below is about 3.5 minutes, so total time blocked is about 2 minutes.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/70570/70570_create.png&quot; width=&quot;800&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="195679">SERVER-18014</key>
            <summary>Dropping a collection can block creating a new collection for an extended time under WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="michael.cahill@mongodb.com">Michael Cahill</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>ET</label>
                    </labels>
                <created>Mon, 13 Apr 2015 18:15:38 +0000</created>
                <updated>Tue, 15 Dec 2015 04:52:14 +0000</updated>
                            <resolved>Mon, 27 Apr 2015 20:46:40 +0000</resolved>
                                    <version>3.0.1</version>
                                    <fixVersion>3.0.3</fixVersion>
                    <fixVersion>3.1.3</fixVersion>
                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="916282" author="xgen-internal-githook" created="Mon, 18 May 2015 02:24:36 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: Don&apos;t try to checkpoint dead handles.&lt;/p&gt;

&lt;p&gt;If a handle is busy when a checkpoint starts (e.g., in the middle of a bulk load), then dead by the time the checkpoint visits it (e.g., a forced drop happens after the checkpoint starts).&lt;/p&gt;

&lt;p&gt;refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: develop&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/c207b64a1f7be98ea65a3ae95407c4da1353498a&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/c207b64a1f7be98ea65a3ae95407c4da1353498a&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="916281" author="xgen-internal-githook" created="Mon, 18 May 2015 02:24:35 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: Add forced drops to test/fops.  We have seen failures where a checkpoint interleaves between a bulk load and a forced drop to try to operate on a dead handle.&lt;/p&gt;

&lt;p&gt;refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: develop&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/80cc29d3531500a529cf044f9419195fe54f9a47&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/80cc29d3531500a529cf044f9419195fe54f9a47&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="898489" author="xgen-internal-githook" created="Mon, 27 Apr 2015 07:42:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;agorrod&apos;, u&apos;name&apos;: u&apos;Alex Gorrod&apos;, u&apos;email&apos;: u&apos;alexg@wiredtiger.com&apos;}
&lt;p&gt;Message: If getting a handle lock only - don&apos;t propogate WT_NOTFOUND.&lt;/p&gt;

&lt;p&gt;It&apos;s expected after the background drop changes.&lt;br/&gt;
Refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: develop&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/5991f88fefb1a5989f9b3633b7cd5c0dc1d57854&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/5991f88fefb1a5989f9b3633b7cd5c0dc1d57854&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="898479" author="xgen-internal-githook" created="Mon, 27 Apr 2015 07:17:35 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: Close block manager handles as soon as a handle is marked dead.&lt;/p&gt;

&lt;p&gt;refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: develop&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/f1d1d01d0759263b8f18719f63f08696a15e9f91&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/f1d1d01d0759263b8f18719f63f08696a15e9f91&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="898478" author="xgen-internal-githook" created="Mon, 27 Apr 2015 07:17:34 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: Discard trees from cache in the background.&lt;/p&gt;

&lt;p&gt;We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles).  If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed.&lt;/p&gt;

&lt;p&gt;With this change, handles are marked &quot;dead&quot;, and readers will no longer use them.  The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on.&lt;/p&gt;

&lt;p&gt;refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17907&quot; title=&quot;B-tree eviction blocks access to collection for extended period under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17907&quot;&gt;&lt;del&gt;SERVER-17907&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: develop&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="896537" author="bruce.lucas@10gen.com" created="Thu, 23 Apr 2015 18:46:08 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;, that build does the trick. The drop command doesn&apos;t show up at all in the stack trace sample (because it happened too quickly), a short time later the sweep server is seen spending a bunch of time in __wt_conn_btree_sync_and_close (most of that in free, because this was Windows...), and things like listDatabases and createCollection are not blocked. Ship it! &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="888783" author="xgen-internal-githook" created="Thu, 16 Apr 2015 07:37:46 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;michaelcahill&apos;, u&apos;name&apos;: u&apos;Michael Cahill&apos;, u&apos;email&apos;: u&apos;michael.cahill@mongodb.com&apos;}
&lt;p&gt;Message: Discard trees from cache in the background.&lt;/p&gt;

&lt;p&gt;We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles).  If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed.&lt;/p&gt;

&lt;p&gt;With this change, handles are marked &quot;dead&quot;, and readers will no longer use them.  The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on.&lt;/p&gt;

&lt;p&gt;refs &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17907&quot; title=&quot;B-tree eviction blocks access to collection for extended period under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-17907&quot;&gt;&lt;del&gt;SERVER-17907&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-18014&quot; title=&quot;Dropping a collection can block creating a new collection for an extended time under WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-18014&quot;&gt;&lt;del&gt;SERVER-18014&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: tree-discard-background&lt;br/&gt;
&lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="194477">SERVER-17907</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="70570" name="create.png" size="507654" author="bruce.lucas@mongodb.com" created="Mon, 13 Apr 2015 18:15:38 +0000"/>
                            <attachment id="70687" name="freeing.png" size="366804" author="bruce.lucas@mongodb.com" created="Tue, 14 Apr 2015 11:38:16 +0000"/>
                            <attachment id="70700" name="linux-02.png" size="168578" author="bruce.lucas@mongodb.com" created="Tue, 14 Apr 2015 12:17:31 +0000"/>
                            <attachment id="70701" name="linux-03.png" size="201577" author="bruce.lucas@mongodb.com" created="Tue, 14 Apr 2015 12:17:31 +0000"/>
                            <attachment id="71197" name="patch-56-sync-drop.png" size="346187" author="bruce.lucas@mongodb.com" created="Fri, 17 Apr 2015 13:47:38 +0000"/>
                            <attachment id="70688" name="procdump.html" size="453164" author="bruce.lucas@mongodb.com" created="Tue, 14 Apr 2015 11:38:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12451" key="com.atlassian.jira.plugin.system.customfieldtypes:multiversion">
                        <customfieldname>Backport Completed</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="15290">3.0.3</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 14 Apr 2015 04:10:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 39 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WRITING-1301'>WRITING-1301</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.prochazka@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 39 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl8gn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs81nb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11858"><![CDATA[Completed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsg48n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>