<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:15:41 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7821] MongoS blocks all requests to sharded collection</title>
                <link>https://jira.mongodb.org/browse/SERVER-7821</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Three times on the last 48 hours, all our mongoS were deadlocked on serving requests to sharded collection.&lt;br/&gt;
Using out-of-prod mongoS to a sharded collection was blocked to.&lt;/p&gt;

&lt;p&gt;During the lock, it was possible to : use db.currentOp() and  show collection on the sharded database.&lt;/p&gt;

&lt;p&gt;Restarting all mongoS was the only way to get out of this.&lt;/p&gt;

&lt;p&gt;In attachment, a cleaned log of one mongoS during the last failure.&lt;/p&gt;</description>
                <environment>Linux Centos5/6</environment>
        <key id="58172">SERVER-7821</key>
            <summary>MongoS blocks all requests to sharded collection</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.mongodb.org/images/icons/priorities/blocker.svg">Blocker - P1</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="kamaradclimber">Gr&#233;goire Seux</reporter>
                        <labels>
                            <label>mongos</label>
                    </labels>
                <created>Mon, 3 Dec 2012 14:38:00 +0000</created>
                <updated>Fri, 8 Mar 2013 15:55:54 +0000</updated>
                            <resolved>Sun, 24 Feb 2013 23:16:14 +0000</resolved>
                                    <version>2.2.1</version>
                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="214681" author="david.hows" created="Fri, 14 Dec 2012 05:23:45 +0000"  >&lt;p&gt;Hi Kelbert,&lt;/p&gt;

&lt;p&gt;Thanks for all the details.&lt;/p&gt;

&lt;p&gt;I&apos;ve been looking into your MMS instance and have found a few things that need your attention.&lt;/p&gt;

&lt;p&gt;I cannot find the 2 SV nodes for set6 in MMS, one of the two SV nodes appears to be the primary and there is currently no replication data for the two NY nodes.&lt;/p&gt;

&lt;p&gt;From what i&apos;ve found in your logs it appears that the migrations from Shard4 to Shard6 are failing when attempting to confirm that the migration was successfully written to a majority of its secondaries, in this case this means having written it to both the NY and SV sites and that migration is failing on this replication timeout.&lt;/p&gt;

&lt;p&gt;My suggestion for now would be to look at enabling &lt;a href=&quot;http://docs.mongodb.org/manual/core/sharding-internals/#chunk-migration&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secondaryThrottle&lt;/a&gt; on your balancer by issuing the following commands:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;use config;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.settings.update ({ &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;_id&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; : &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;balancer&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}, {$set:{_secondaryThrottle&quot; : &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; }, &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;secondaryThrottle will change the migration settings such that when migrating a chunk your system should only attempt to replicate to one node rather than a majority. &lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="213888" author="k.hodin@criteo.com" created="Thu, 13 Dec 2012 10:50:51 +0000"  >&lt;p&gt;David, &lt;/p&gt;

&lt;p&gt;My answers :&lt;/p&gt;

&lt;p&gt;Do you have MMS installed? &lt;br/&gt;
Yes. &lt;a href=&quot;https://mms.10gen.com/host/list/4f8d732587d1d86fa8b99c12&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mms.10gen.com/host/list/4f8d732587d1d86fa8b99c12&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Is there any replication lag between your nodes in Shard6? &lt;br/&gt;
No lag in this shard.&lt;/p&gt;

&lt;p&gt;Can you give me a little background on your clusters setup and layout? &lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;8 shards&lt;/li&gt;
	&lt;li&gt;each shard is a 4 servers replicaset (2 DC, 2 host per DC)&lt;/li&gt;
	&lt;li&gt;all in mongodb 2.2.2&lt;/li&gt;
	&lt;li&gt;collection that fails contains 11 469 docs only.&lt;/li&gt;
	&lt;li&gt;collection is blocked (no find nor update) every morning until we restart shard4 mongod&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Can you provide the output of sh.status()?&lt;br/&gt;
File attached. &lt;br/&gt;
Problem occured before adding 8th shard.&lt;/p&gt;</comment>
                            <comment id="213612" author="david.hows" created="Thu, 13 Dec 2012 00:58:09 +0000"  >&lt;p&gt;Hi Kelbert,&lt;/p&gt;

&lt;p&gt;From 06:46:37 till 06:47:08 your shards were in the critical section of migration, which subsequently failed by timing out after 30 seconds. This is likely the cause of the unresponsive behaviour at that time.&lt;/p&gt;

&lt;p&gt;The migration appears to have failed as it could not adequately replicate the full migration to the three secondaries it attempted to ensure replication on.&lt;/p&gt;

&lt;p&gt;Do you have MMS installed? Is there any replication lag between your nodes in Shard6? Can you give me a little background on your clusters setup and layout? Can you provide the output of &lt;tt&gt;sh.status()&lt;/tt&gt;?&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;
</comment>
                            <comment id="213211" author="k.hodin@criteo.com" created="Wed, 12 Dec 2012 17:20:41 +0000"  >&lt;p&gt;Hi David,&lt;/p&gt;

&lt;p&gt;This issue happened again this morning, no update nor find could be done on counters.statistics collection.&lt;br/&gt;
You can find attached logs of this morning of these 2 shards, issue starts at Wed Dec 12 06:47:09.&lt;/p&gt;

&lt;p&gt;Kl&#233;bert&lt;/p&gt;</comment>
                            <comment id="212666" author="david.hows" created="Wed, 12 Dec 2012 00:40:06 +0000"  >&lt;p&gt;Hi Kelbert, &lt;/p&gt;

&lt;p&gt;It looks like that freeze is due to to the system being in the &quot;critical section&quot; from the small log snippet you have sent I can see your system exiting the critical section at &lt;tt&gt;05:50:48&lt;/tt&gt; but cannot see when we entered, could you add more log details?&lt;/p&gt;

&lt;p&gt;From the logs i can also see that the migration failed as it was not accepted by the &quot;to&quot; shard.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Tue Dec 11 05:50:48 [conn370166] moveChunk migrate commit not accepted by TO-shard: { active: true, ns: &quot;counters.statistics&quot;, from: &quot;shard4/mdbcis4-01-ny.criteo.prod:27021,mdbcis4-01-sv.criteo.prod:27021,mdbcis4-02-ny.criteo.prod:27021,mdbcis4-02-sv.criteo.prod:27021&quot;, min: { _id: &quot;7f8f7d68-5ad5-49e6-92a6-aff7181d6c7e&quot; }, max: { _id: &quot;bfbef656-d8c3-4742-89fc-716a4b3d4c54&quot; }, shardKeyPattern: { _id: 1 }, state: &quot;fail&quot;, errmsg: &quot;&quot;, counts: { cloned: 19935, clonedBytes: 8525712, catchup: 0, steady: 0 }, errmsg: &quot;&quot;, ok: 0.0 } resetting shard version to: 0|0||000000000000000000000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Can you attach logs from Shard4 and Shard6 around this time? As these appear to be the &quot;to&quot; and &quot;from&quot; shards.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="211956" author="k.hodin@criteo.com" created="Tue, 11 Dec 2012 09:57:36 +0000"  >&lt;p&gt;Hi David,&lt;/p&gt;

&lt;p&gt;It happened again this morning at 5:50 am (more info in log file attached).&lt;/p&gt;</comment>
                            <comment id="211906" author="kamaradclimber" created="Tue, 11 Dec 2012 08:08:44 +0000"  >&lt;p&gt;Hello David,&lt;/p&gt;

&lt;p&gt;we don&apos;t have freeze anymore. Since the upgrade (2.2.1 =&amp;gt; 2.2.2) the issue has shifted to being unable to update some collections. Changelog attached&lt;/p&gt;</comment>
                            <comment id="211822" author="david.hows" created="Tue, 11 Dec 2012 05:15:42 +0000"  >&lt;p&gt;Hi Kelbert, Gregorie,&lt;/p&gt;

&lt;p&gt;I can see that there is a large migration which subsequently fails and this looks to have changed your chunk version.&lt;/p&gt;

&lt;p&gt;Restarting the replica set would have fixed the issue as this would have cleared the mongod&apos;s internal cache.&lt;/p&gt;

&lt;p&gt;Do you stil get these freezes subsequent to the restart?&lt;/p&gt;

&lt;p&gt;Would you be able to attach the contents of the &lt;tt&gt;changelog&lt;/tt&gt; collection in the &lt;tt&gt;config&lt;/tt&gt; database on your mongos?&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="210889" author="k.hodin@criteo.com" created="Mon, 10 Dec 2012 09:26:13 +0000"  >&lt;p&gt;This issue seems to be linked to one mongod log message : &quot; warning: aborted moveChunk because official version less than mine? &quot;&lt;/p&gt;

&lt;p&gt;It first appears at line 14 862.&lt;/p&gt;</comment>
                            <comment id="210863" author="kamaradclimber" created="Mon, 10 Dec 2012 08:39:50 +0000"  >&lt;p&gt;upgrade moved the issue. Now some collections have trouble updating documents. Restart of the replicaset fix the issue&lt;/p&gt;</comment>
                            <comment id="205981" author="kamaradclimber" created="Tue, 4 Dec 2012 09:01:05 +0000"  >&lt;p&gt;Our dba observed that it is probably linked to another issue on mongoD : &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7034&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-7034&lt;/a&gt; because it happens on one mongoD roughly at the same time than this issue.&lt;br/&gt;
Unlike Spencer said, this happens very frequently and cannot be considered as acceptable for our production environment.&lt;/p&gt;

&lt;p&gt;We will upgrade to 2.2.2 today and see what the result is.&lt;/p&gt;

</comment>
                            <comment id="205964" author="kamaradclimber" created="Tue, 4 Dec 2012 08:23:15 +0000"  >&lt;p&gt;Here is the log from this night. The same behavior occured aroudn 2:35am&lt;/p&gt;</comment>
                            <comment id="205960" author="kamaradclimber" created="Tue, 4 Dec 2012 08:14:00 +0000"  >&lt;p&gt;Hello David,&lt;/p&gt;

&lt;p&gt;we use mongo 2.2.1.&lt;/p&gt;

&lt;p&gt;findOne on sharded collection hangs for more than 5 minutes (and judging by mms, indefinitely).&lt;br/&gt;
findOne on unsharded database works as usual&lt;/p&gt;

&lt;p&gt;I&apos;ve tried db.currentOp on the last time it occurs (without saving output) but I&apos;ll try to get next time.&lt;/p&gt;</comment>
                            <comment id="205803" author="david.hows" created="Tue, 4 Dec 2012 03:00:59 +0000"  >&lt;p&gt;Hi Gr&#233;goire,&lt;/p&gt;

&lt;p&gt;Can you confirm which version of mongo you are using?&lt;/p&gt;

&lt;p&gt;Would you be able to post a little more context around the log you have provided? The lines in the currently log dont show much.&lt;/p&gt;

&lt;p&gt;Could you attach the output of &lt;tt&gt;db.currentOp()&lt;/tt&gt; from the time when the system is unresponsive?&lt;/p&gt;

&lt;p&gt;What happens if you try to do a simple findOne() on your system when this behaviour occurs? Does it just hang? Does the client get an error? &lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="205216" author="kamaradclimber" created="Mon, 3 Dec 2012 14:39:13 +0000"  >&lt;p&gt;sed s/&lt;span class=&quot;error&quot;&gt;&amp;#91;dead&amp;#93;&lt;/span&gt;lock/unresponsive bahvior/&lt;br/&gt;
because I don&apos;t know if it is a dead lock or something else.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="59270">SERVER-7922</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="22373" name="changelog_us" size="11339074" author="kamaradclimber" created="Tue, 11 Dec 2012 08:09:01 +0000"/>
                            <attachment id="22132" name="deadlock.png" size="65214" author="kamaradclimber" created="Mon, 3 Dec 2012 14:38:00 +0000"/>
                            <attachment id="22133" name="lock.log" size="5228" author="kamaradclimber" created="Mon, 3 Dec 2012 14:38:00 +0000"/>
                            <attachment id="22153" name="mongoS.gz" size="48230034" author="kamaradclimber" created="Tue, 4 Dec 2012 08:23:15 +0000"/>
                            <attachment id="22440" name="mongod-shard4-shard6-logs.zip" size="1447231" author="k.hodin@criteo.com" created="Wed, 12 Dec 2012 17:20:41 +0000"/>
                            <attachment id="22374" name="moveChunk_abort_ version_issue_20121211.txt" size="64338" author="k.hodin@criteo.com" created="Tue, 11 Dec 2012 09:57:36 +0000"/>
                            <attachment id="22462" name="sh_status.out" size="6125" author="k.hodin@criteo.com" created="Thu, 13 Dec 2012 10:50:51 +0000"/>
                            <attachment id="22341" name="shard4_log_extract.zip" size="465542" author="k.hodin@criteo.com" created="Mon, 10 Dec 2012 09:26:13 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 4 Dec 2012 03:00:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 9 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 9 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>kamaradclimber</customfieldvalue>
            <customfieldvalue>k.hodin@criteo.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnfv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrkvun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>35626</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht05hj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>