<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:04:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-23935] Disable oplog sampling in queryable backup mode</title>
                <link>https://jira.mongodb.org/browse/SERVER-23935</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When running with &lt;tt&gt;&amp;#45;&amp;#45;queryableBackupMode&lt;/tt&gt;, &lt;tt&gt;WiredTigerKVEngine::initRsOplogBackgroundThread()&lt;/tt&gt; should not start a thread, and therefore not perform oplog sampling.&lt;/p&gt;

&lt;p&gt;In addition, we should consider adding a separate configuration option to disable this oplog sampling without needing to be running with &lt;tt&gt;&amp;#45;&amp;#45;queryableBackupMode&lt;/tt&gt;.&lt;/p&gt;

&lt;h4&gt;&lt;a name=&quot;OriginalDescription&quot;&gt;&lt;/a&gt;Original Description&lt;/h4&gt;

&lt;p&gt;In version 3.1.8 mongo added a new feature: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19551&quot; title=&quot;Keep &amp;quot;milestones&amp;quot; against WT oplog to efficiently remove old records&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19551&quot;&gt;&lt;del&gt;SERVER-19551&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On startup WiredTiger now needs to read values from the oplog to figure out where to place the milestones:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-04-26T15:42:22.066+0000 I STORAGE  [initandlisten] The size storer reports that the oplog contains 19279531 records totaling to 53948096450 bytes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-04-26T15:42:22.410+0000 I STORAGE  [initandlisten] Sampling from the oplog between Apr 23 22:02:46:71 and Apr 26 14:10:01:d1 to determine where to place markers for truncation&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-04-26T15:42:22.410+0000 I STORAGE  [initandlisten] Taking 1004 samples and assuming that each section of oplog contains approximately 191863 records totaling to 536872169 bytes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This process is fine if all of the data is stored locally on disk. The problem is when you are using external storage (for us, Amazon EBS).  The sampling from cold EBS is taking somewhere between 20 and 30 minutes leading to incredibly slow startup times.   &lt;/p&gt;

&lt;p&gt;Is there a good way to speed up this process / bring startup times back to 3.1.7 levels for EBS backed instances?&lt;/p&gt;</description>
                <environment></environment>
        <key id="282520">SERVER-23935</key>
            <summary>Disable oplog sampling in queryable backup mode</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="neha.khatri">Neha Khatri</assignee>
                                    <reporter username="paetling@gmail.com">Alex Etling</reporter>
                        <labels>
                            <label>neweng</label>
                    </labels>
                <created>Tue, 26 Apr 2016 17:19:06 +0000</created>
                <updated>Wed, 2 May 2018 03:40:11 +0000</updated>
                            <resolved>Thu, 19 Oct 2017 14:09:11 +0000</resolved>
                                    <version>3.1.8</version>
                                    <fixVersion>3.4.15</fixVersion>
                    <fixVersion>3.6.0-rc1</fixVersion>
                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>14</watches>
                                                                                                                <comments>
                            <comment id="1879920" author="xgen-internal-githook" created="Wed, 2 May 2018 03:15:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;neha.khatri@mongodb.com&apos;, &apos;name&apos;: &apos;nehakhatri5&apos;, &apos;username&apos;: &apos;nehakhatri5&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23935&quot; title=&quot;Disable oplog sampling in queryable backup mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23935&quot;&gt;&lt;del&gt;SERVER-23935&lt;/del&gt;&lt;/a&gt; Disable oplog sampling in queryable backup mode&lt;/p&gt;

&lt;p&gt;In queryable backup mode the oplog truncation would never occur. Hence oplog sampling&lt;br/&gt;
is disabled this mode.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 1c96c3561dda50fc3ba6d98decef1c0d3c9f60df)&lt;br/&gt;
Branch: v3.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ff63eee6bfba4d0cd4c1eaf1a95e5ada19f26afa&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ff63eee6bfba4d0cd4c1eaf1a95e5ada19f26afa&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1837650" author="asya" created="Sun, 18 Mar 2018 07:41:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt; I&apos;ve requested a backport in BACKPORT-1888 - it will be considered by the team during the next triage session.&lt;/p&gt;</comment>
                            <comment id="1836050" author="bartle" created="Fri, 16 Mar 2018 05:14:49 +0000"  >&lt;p&gt;Would it be possible to backport this to 3.4?  I believe it should be a fairly trivial backport.&lt;/p&gt;</comment>
                            <comment id="1703735" author="neha.khatri" created="Thu, 19 Oct 2017 14:09:11 +0000"  >&lt;p&gt;A code change has been pushed in master to prohibit the oplog sampliing in queryableBackupMode.&lt;/p&gt;</comment>
                            <comment id="1703728" author="xgen-internal-githook" created="Thu, 19 Oct 2017 13:51:15 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;neha.khatri@mongodb.com&apos;, &apos;name&apos;: &apos;nehakhatri5&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23935&quot; title=&quot;Disable oplog sampling in queryable backup mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23935&quot;&gt;&lt;del&gt;SERVER-23935&lt;/del&gt;&lt;/a&gt; Disable oplog sampling in queryable backup mode&lt;/p&gt;

&lt;p&gt;In queryable backup mode the oplog truncation would never occur. Hence oplog sampling&lt;br/&gt;
is disabled this mode.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1c96c3561dda50fc3ba6d98decef1c0d3c9f60df&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1c96c3561dda50fc3ba6d98decef1c0d3c9f60df&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1260928" author="paetling@gmail.com" created="Tue, 10 May 2016 16:50:22 +0000"  >&lt;p&gt;Max, &lt;br/&gt;
     This is awesome!  I look forward to when this is implemented.  &lt;/p&gt;

&lt;p&gt;     As far as documentation goes, I think this might be a semi-common use case. It is probably worth documenting the `--queryableBackupMode` option and how it can be used to prevent sampling.  &lt;/p&gt;

&lt;p&gt;Alex&lt;/p&gt;</comment>
                            <comment id="1260769" author="max.hirschhorn@10gen.com" created="Tue, 10 May 2016 15:03:31 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paetling%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;paetling@gmail.com&quot;&gt;paetling@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I wanted to give you an update with regard to some of the internal discussion we&apos;ve been having around this ticket. While I&apos;m not familiar enough with cold EBS to estimate how long ~1000 random walks in a WiredTiger B-tree should take, 20-30 minutes seems like an undesirably long time.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20368&quot; title=&quot;Only start a background thread for the active oplog in WT&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20368&quot;&gt;&lt;del&gt;SERVER-20368&lt;/del&gt;&lt;/a&gt; would have made it possible to avoid sampling the oplog when the mongod is started up as a stand-alone node. The intention behind &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20368&quot; title=&quot;Only start a background thread for the active oplog in WT&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20368&quot;&gt;&lt;del&gt;SERVER-20368&lt;/del&gt;&lt;/a&gt; was to provide users with a way to still be able to perform an update that changed the size of an oplog entry because that explicitly became forbidden as part of the design for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19551&quot; title=&quot;Keep &amp;quot;milestones&amp;quot; against WT oplog to efficiently remove old records&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19551&quot;&gt;&lt;del&gt;SERVER-19551&lt;/del&gt;&lt;/a&gt;. However, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20529&quot; title=&quot;WiredTiger allows capped collection objects to grow&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20529&quot;&gt;&lt;del&gt;SERVER-20529&lt;/del&gt;&lt;/a&gt; made it an error to try and change the size of a document in any capped collection, so &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20368&quot; title=&quot;Only start a background thread for the active oplog in WT&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20368&quot;&gt;&lt;del&gt;SERVER-20368&lt;/del&gt;&lt;/a&gt; was thought to be unnecessary.&lt;/p&gt;

&lt;p&gt;Given that your use case is read-only in order to verify the snapshot, I agree it makes sense not to have to sample from the oplog on startup because a truncation of the oplog shouldn&apos;t ever occur. We are therefore going to transform this ticket into a feature request so that &lt;tt&gt;WiredTigerKVEngine::initRsOplogBackgroundThread()&lt;/tt&gt; skips starting a thread and thus skips sampling the oplog when the undocumented &lt;tt&gt;&amp;#45;&amp;#45;queryableBackupMode&lt;/tt&gt; option is specified (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-593&quot; title=&quot;read only mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-593&quot;&gt;&lt;del&gt;SERVER-593&lt;/del&gt;&lt;/a&gt;) in versions 3.3+. While I don&apos;t think we&apos;d consider backporting this change to 3.2 under a different server parameter, it would be possible for you to use a pre-3.4 release (i.e. a 3.3 development version) to check that your snapshots are intact.&lt;/p&gt;

&lt;p&gt;Let us know if you have any further questions or concerns.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Max&lt;/p&gt;</comment>
                            <comment id="1260428" author="ramon.fernandez" created="Tue, 10 May 2016 11:24:15 +0000"  >&lt;p&gt;Thanks for your report &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paetling%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;paetling@gmail.com&quot;&gt;paetling@gmail.com&lt;/a&gt;, we&apos;re dispatching it internally for consideration.&lt;/p&gt;</comment>
                            <comment id="1246992" author="travis@gamechanger.io" created="Tue, 26 Apr 2016 17:50:38 +0000"  >&lt;p&gt;To clarify, this is an issue primarily when attempting to start a mongod off of an uninitialized EBS volume, e.g. one that was just restored from a snapshot and has not yet read its data files in from S3. Details here: &lt;a href=&quot;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;On an already-initialized EBS volume, startup times are only a second or two slower than pre-3.1.7 speeds in our experience (50GB oplog).&lt;/p&gt;

&lt;p&gt;Our use case is attempting to verify that an EBS snapshot contains a valid Mongo data directory. To do this, we restore the snapshot to a volume, mount the volume, start a mongod against the volume&apos;s data directory, and attempt to query our collections.&lt;/p&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="14340"><![CDATA[v3.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000VEHxFIAX]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 26 Apr 2016 17:50:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 41 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 41 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>paetling@gmail.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>bartle</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>neha.khatri</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>travis@gamechanger.io</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk973:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9sbz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1925">Storage 2017-10-23</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs5fzj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>