<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:19:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49233] Introduce a flag to toggle the logic for bumping collection&apos;s major version during split</title>
                <link>https://jira.mongodb.org/browse/SERVER-49233</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;div class=&quot;panel&quot; style=&quot;background-color: #eeeeee;border-color: #cccccc;border-width: 1px;&quot;&gt;&lt;div class=&quot;panelHeader&quot; style=&quot;border-bottom-width: 1px;border-bottom-color: #cccccc;background-color: #6cb33f;&quot;&gt;&lt;b&gt;Issue Status as of July 12 2020&lt;/b&gt;&lt;/div&gt;&lt;div class=&quot;panelContent&quot; style=&quot;background-color: #eeeeee;&quot;&gt;
&lt;p&gt;&lt;b&gt;ISSUE SUMMARY&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;This fix introduces a global flag named &lt;tt&gt;incrementChunkMajorVersionOnChunkSplits&lt;/tt&gt; which is used to choose between two algorithms for updating a collection&#8217;s routing table after performing a chunk split in a sharded cluster. Chunk split operations and their corresponding changes to a collection&#8217;s routing table causes mongoses to refresh their routing table caches, requiring resources and briefly stalling CRUD operations, which can increase CPU load and operation latency.&lt;/p&gt;

&lt;p&gt;Before 3.6.15 and 4.0.13, the algorithm worked well for shard keys with uniformly distributed values, but led to performance problems for cases where shard key values are incrementing monotonically. A change introduced by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;del&gt;SERVER-41480&lt;/del&gt;&lt;/a&gt; changed the algorithm to fix that edge case.&lt;/p&gt;

&lt;p&gt;The changes introduced with this ticket cause the old algorithm to be used by default, but allows the customer to opt into the behavior introduced by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;del&gt;SERVER-41480&lt;/del&gt;&lt;/a&gt; if needed.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;USER IMPACT&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Before 3.6.15 and 4.0.13, the mongos&#8217;s cached version of the routing table for a collection is not updated until a chunk is moved or until the mongos succeeds in splitting a chunk. This means that, during inserts to a sharded collection with monotonically increasing shard key values, if the chunk with the max shard key is split by one mongos, other mongoses may try and fail to split this chunk because of out-of-date routing tables in their caches. These continual failed split attempts can result in a high CPU load on the shard containing this chunk.&lt;/p&gt;

&lt;p&gt;In versions 3.6.15 through 3.6.18, 4.0.13 through 4.0.19 and 4.2.2 through 4.2.8 during inserts to documents with uniformly distributed shard key values, due to the change made by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;del&gt;SERVER-41480&lt;/del&gt;&lt;/a&gt;, mongoses are forced to update their routing table caches more often than necessary, causing CRUD operations to be blocked while routing table caches are being updated.&lt;/p&gt;

&lt;p&gt;From 4.2 onwards, the splitting is triggered by shards, meaning that the problem that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;del&gt;SERVER-41480&lt;/del&gt;&lt;/a&gt; was designed to address is no longer present. However, the changes it introduced can still cause routing table caches to be updated more often than necessary.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;SUGGESTED ACTION&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;For 3.6 and 4.0 installations, upgrade to the latest minor version which supports the flag in mongod. If your deployment inserts documents with monotonically increasing or decreasing shard keys and you observe high load due to failed chunk split attempts, set the flag to &#8220;true&#8221;, otherwise no additional action is required. The flag can be set to true in the mongod configuration file, or with &lt;br/&gt;
 &lt;tt&gt;--setParameter incrementChunkMajorVersionOnChunkSplit=true&lt;/tt&gt;&lt;br/&gt;
 on the command line when starting mongod.&lt;/p&gt;

&lt;p&gt;For 4.2 installations, upgrade to 4.2.9. No further action is needed.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;AFFECTED VERSIONS&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;This issue affects versions 3.6.15 through 3.6.18, 4.0.13 through 4.0.19 and 4.2.2 through 4.2.8.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;FIX VERSION&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;The flag is available in versions 3.6.19 and 4.0.20. In 4.2.9 both uniform and monotonic shard key distributions are supported and so no flag is necessary.&lt;/p&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;h6&gt;&lt;a name=&quot;OriginalDescription&quot;&gt;&lt;/a&gt;Original Description&lt;/h6&gt;

&lt;p&gt;This ticket is to introduce a flag, which toggles the logic for bumping the collection&apos;s major version during split.&lt;/p&gt;

&lt;p&gt;The flag should only be meaningful for the Config Server and should default to OFF (meaning the major version should not be bumped by default).&lt;/p&gt;

&lt;p&gt;This flag should be present in 3.6, 4.0 and 4.2. &lt;del&gt;For 4.4, since the auto-splitter is running on the shards, this flag should be removed, along with the aforementioned logic for bumping the major version.&lt;/del&gt;&lt;/p&gt;

&lt;p&gt;Tl;dr:&lt;br/&gt;
 As part of a fix for a high CPU utilisation by the auto-splitter when writes are happening to the extreme chunks, we chose a solution, which erred on the side of correctness, with the reasoning that on most systems auto-splits are happening rarely and are not happening at the same time across all shards.&lt;/p&gt;

&lt;p&gt;Under this solution, the collection&apos;s major version would be incremented only if a split happens at the shard, whose shardVersion == collectionVersion. Therefore, unless a split happens on this shard, all the other shards will just catch-up to its shard version and will then stop. This will repeat when the shard containing the collection version performs a split again, so the idea was that this will be amortised across the number of shards.&lt;/p&gt;

&lt;p&gt;However, this logic seems to be causing more harm than good in the case of almost uniform writes across all chunks. If it is the case that all shards are doing splits almost in unison, under this fix there will constantly be a bump in the collection version, which means constant stalls due to StaleShardVersion.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1399067">SERVER-49233</key>
            <summary>Introduce a flag to toggle the logic for bumping collection&apos;s major version during split</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="matthew.saltz@mongodb.com">Matthew Saltz</assignee>
                                    <reporter username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</reporter>
                        <labels>
                    </labels>
                <created>Wed, 1 Jul 2020 17:58:57 +0000</created>
                <updated>Sun, 29 Oct 2023 22:06:14 +0000</updated>
                            <resolved>Mon, 20 Jul 2020 16:10:39 +0000</resolved>
                                                    <fixVersion>4.0.20</fixVersion>
                    <fixVersion>3.6.19</fixVersion>
                    <fixVersion>4.2.9</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="3317803" author="xgen-internal-githook" created="Mon, 3 Aug 2020 15:58:02 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Saltz&apos;, &apos;email&apos;: &apos;matthew.saltz@mongodb.com&apos;, &apos;username&apos;: &apos;saltzm&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49233&quot; title=&quot;Introduce a flag to toggle the logic for bumping collection&amp;#39;s major version during split&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49233&quot;&gt;&lt;del&gt;SERVER-49233&lt;/del&gt;&lt;/a&gt; Introduce a flag to toggle the logic for bumping collection&apos;s major version during split&lt;br/&gt;
Branch: v4.2&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/96006491b33820538a5dd850fd0df9fdb6aaa1c9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/96006491b33820538a5dd850fd0df9fdb6aaa1c9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3284568" author="xgen-internal-githook" created="Tue, 14 Jul 2020 16:03:45 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Saltz&apos;, &apos;email&apos;: &apos;matthew.saltz@mongodb.com&apos;, &apos;username&apos;: &apos;saltzm&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49233&quot; title=&quot;Introduce a flag to toggle the logic for bumping collection&amp;#39;s major version during split&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49233&quot;&gt;&lt;del&gt;SERVER-49233&lt;/del&gt;&lt;/a&gt; Introduce a flag to toggle the logic for bumping collection&apos;s major version during split&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/b2d22916f3f28b1534ce3613685713ca027827e7&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/b2d22916f3f28b1534ce3613685713ca027827e7&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3276052" author="xgen-internal-githook" created="Wed, 8 Jul 2020 20:37:45 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Matthew Saltz&apos;, &apos;email&apos;: &apos;matthew.saltz@mongodb.com&apos;, &apos;username&apos;: &apos;saltzm&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49233&quot; title=&quot;Introduce a flag to toggle the logic for bumping collection&amp;#39;s major version during split&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49233&quot;&gt;&lt;del&gt;SERVER-49233&lt;/del&gt;&lt;/a&gt; Introduce a flag to toggle the logic for bumping collection&apos;s major version during split&lt;br/&gt;
Branch: v3.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f0253e7ba95430c7e100b4ab5f76dd786f9f9a3d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f0253e7ba95430c7e100b4ab5f76dd786f9f9a3d&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="1414837">DOCS-13774</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="787957">SERVER-41480</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1406469">SERVER-49433</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1408012">SERVER-49479</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16775"><![CDATA[v4.2]]></customfieldvalue>
    <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10011"><![CDATA[Minor Change]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000npOkgQAE, 5002K00000obr2gQAA, 5002K00000ocpTlQAI, 5002K00000pl0NlQAI, 5002K00000pESvEQAW, 5002K00000pGWxZQAW, 5002K00000xCirAQAS]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 1 Jul 2020 18:13:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 27 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>This introduces a new server parameter incrementChunkMajorVersionOnChunkSplits into 3.6 and 4.0 that affects the frequency of sharding metadata refreshes caused by autosplitting. Its default value is false and should only be modified in rare cases (see the ticket description for more details). The flag effectively makes the behavior introduced by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;strike&gt;SERVER-41480&lt;/strike&gt;&lt;/a&gt; opt-in, so customers who previously had benefited from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-41480&quot; title=&quot;Increment collection major version on chunk splits/merges&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-41480&quot;&gt;&lt;strike&gt;SERVER-41480&lt;/strike&gt;&lt;/a&gt; will want to set the flag to true.</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 27 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>41.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.saltz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxt2sv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxfzlb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3954">Sharding 2020-07-13</customfieldvalue>
    <customfieldvalue id="4135">Sharding 2020-07-27</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16943"><![CDATA[Cloud]]></customfieldvalue>
    <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    <customfieldvalue key="16946"><![CDATA[Triage and Release]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxsp27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>