<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:16:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-8189] Unstable connectivity in replica sets environment</title>
                <link>https://jira.mongodb.org/browse/SERVER-8189</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello! We have production setup of three nodes in replica sets, elba-mongo1 on one rack (the primary) and elba-mongo2, elba-mongo3 on another (secondaries).&lt;br/&gt;
Periodically in our mongo logs we see something similar to:&lt;/p&gt;

&lt;p&gt;Wed Jan 16 11:59:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; replSet syncing to: elba-mongo1:27017&lt;br/&gt;
Wed Jan 16 11:59:16 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; DBClientCursor::init call() failed&lt;br/&gt;
Wed Jan 16 11:59:16 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info elba-mongo1:27017 is down (or slow to respond): DBClientBase::findN: transport error: elba-mongo1:27017 ns: admin.$cmd query: { replSetHeartbeat: &quot;rs0&quot;, v: 6, pv: 1, checkEmpty: false, from: &quot;elba-mongo3:27017&quot;, $auth: {} }&lt;/p&gt;

&lt;p&gt;After this point there is an election of new primary. elba-mongo1 pinged successfully from both elba-mongo2 and 3, but we can&apos;t connect to elba-mongo1 instance from elba-moongo1 machine using mongo console - it just hangs on &quot;connecting to&quot;:&lt;/p&gt;

&lt;p&gt;Croot@elba-mongo1:~# telnet localhost 27017 &lt;br/&gt;
Trying 127.0.0.1... &lt;br/&gt;
Connected to localhost. &lt;br/&gt;
Escape character is &apos;^]&apos;. &lt;br/&gt;
^] &lt;br/&gt;
telnet&amp;gt; quit &lt;br/&gt;
Connection closed. &lt;br/&gt;
root@elba-mongo1:~# mongo &lt;br/&gt;
MongoDB shell version: 2.2.2 &lt;br/&gt;
connecting to: test&lt;/p&gt;</description>
                <environment>Debian: Linux elba-mongo3 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux</environment>
        <key id="62143">SERVER-8189</key>
            <summary>Unstable connectivity in replica sets environment</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="tad">Tad Marshall</assignee>
                                    <reporter username="gusev_p">Gusev Petr</reporter>
                        <labels>
                    </labels>
                <created>Wed, 16 Jan 2013 14:16:53 +0000</created>
                <updated>Fri, 15 Feb 2013 15:06:30 +0000</updated>
                            <resolved>Tue, 22 Jan 2013 15:24:04 +0000</resolved>
                                    <version>2.2.2</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="246557" author="tad" created="Tue, 22 Jan 2013 15:23:11 +0000"  >&lt;p&gt;Hi Dmitry,&lt;/p&gt;

&lt;p&gt;I had assumed that actually connecting from the shell in order to do something was a goal, but your timing approach is interesting as well.&lt;/p&gt;

&lt;p&gt;I&apos;m glad that you&apos;re up-and-running now.  I&apos;ll close this ticket, and thanks for letting us know the resolution!&lt;/p&gt;

&lt;p&gt;Tad&lt;/p&gt;</comment>
                            <comment id="246552" author="tad" created="Tue, 22 Jan 2013 15:19:32 +0000"  >&lt;p&gt;Hi Gusev,&lt;/p&gt;

&lt;p&gt;Thanks for letting us know what the problem was.  I&apos;m sorry that I didn&apos;t latch onto NUMA myself, since it is right there in the log you posted.  Since you noted connectivity problems and two primaries, I didn&apos;t think to look for alternate explanations.&lt;/p&gt;

&lt;p&gt;Version 2.4 will display any startup warnings from the server when connecting to it from the mongo shell, which will help to raise the visibility of issues such as NUMA.  A proposal I made in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6889&quot; title=&quot;Refuse to start on NUMA hardware if &amp;quot;numactl --interleave=all&amp;quot; is not set, bypass with --overrideNUMAwarning&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6889&quot;&gt;&lt;del&gt;SERVER-6889&lt;/del&gt;&lt;/a&gt; to require an explicit command line switch to ignore the NUMA warning has not garnered any votes and is marked as &quot;features we&apos;re not sure of&quot;.&lt;/p&gt;

&lt;p&gt;Tad&lt;/p&gt;</comment>
                            <comment id="246541" author="demonixed" created="Tue, 22 Jan 2013 15:12:35 +0000"  >&lt;p&gt;Oh, yeah.. That was a typo in command. My mistake... Speaking about omitting &quot;shell&quot; option - I just tried to measure connect time running three commands in a row (one for each node). I thought leaving client connected or not is not so important, isn&apos;t it?&lt;/p&gt;

&lt;p&gt;Nevertheless, we&apos;ve installed numactl last night. Mongod starts without warnings now and works without any issues.&lt;br/&gt;
We&apos;ll be checking logs for a couple of days. If errors come back, I&apos;ll post a comment. And I&apos;ll keep in mind your advices on using serverStatus and currentOp.&lt;/p&gt;</comment>
                            <comment id="246491" author="gusev_p" created="Tue, 22 Jan 2013 14:17:16 +0000"  >&lt;p&gt;As it turns out the root of the problem was NUMA. We didn&apos;t pay much attention to it initially because the symptoms look differently from what have beed described here &lt;a href=&quot;http://www.wireclub.com/development/TqnkQwQ8CxUYTVT90/read&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.wireclub.com/development/TqnkQwQ8CxUYTVT90/read&lt;/a&gt;. But this article&lt;br/&gt;
&lt;a href=&quot;http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture&lt;/a&gt; have greatly clarified NUMA related stuff. The very reason of unstable server response times (which resulted in replication cycles) was out of physical memory on node-0 (one of two our NUMA nodes) and corresponding long paging intervals. &lt;br/&gt;
Thereby maybe it would make sense to make NUMA warning on server startup more strict - it is very unlikely that current implementation of mongo daemon can operate on NUMA architecture without strange and hard to reproduce instabilities.&lt;br/&gt;
Great thanks to Tad Marshall and all mongo team, you guys do a great job!&lt;/p&gt;</comment>
                            <comment id="245531" author="tad" created="Mon, 21 Jan 2013 12:34:00 +0000"  >&lt;p&gt;If this is how you spelled &quot;prompt&quot; (sic. &quot;propmt&quot;) then you just set a new variable and didn&apos;t change the actual prompt.  Changing the prompt to a fixed string is intended to sidestep any issues related to prompt negotiation.  Setting the &quot;prompt&quot; variable will accomplish this.  Setting &quot;propmt&quot; just sets a variable; not one that means anything to the shell.&lt;/p&gt;

&lt;p&gt;You omitted the &quot;-&lt;del&gt;shell&quot; option in the examples above ... is this an actual log, or are you retyping text from another source?  You need &quot;&lt;/del&gt;&lt;del&gt;shell&quot; to make the shell continue running after it executes the &quot;&lt;/del&gt;-eval&quot; option.&lt;/p&gt;

&lt;p&gt;If you have a working connection from elba-mongo2 to elba-mongo1, you should be able to use serverStatus and currentOp commands to see what state it is in.&lt;/p&gt;

&lt;p&gt;Are your ulimit settings sufficient to allow the number of connections you are trying to have?&lt;/p&gt;</comment>
                            <comment id="245498" author="demonixed" created="Mon, 21 Jan 2013 11:29:15 +0000"  >&lt;p&gt;Hi Tad! I&apos;m another Petr&apos;s co-worker &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; and also one of the admins of datacenter, where above servers are. &lt;br/&gt;
We looked carefully, what&apos;s happening on server when node stop responding. Here&apos;s what we found at this time.&lt;/p&gt;

&lt;p&gt;Mongod continues to accept new connections both from local and remote nodes (proven by telnet), no packet loss while nodes pinging each other but&lt;br/&gt;
mongo console can&apos;t connect to local instance even with custom prompt:&lt;/p&gt;

&lt;p&gt;root@elba-mongo1:~#  time mongo --eval &quot;propmt=&apos;&amp;gt;&apos;&quot; --host elba-mongo1&lt;br/&gt;
MongoDB shell version: 2.2.2&lt;br/&gt;
connecting to: elba-mongo1:27017/test&lt;br/&gt;
&amp;gt;&lt;/p&gt;

&lt;p&gt;real    1m4.098s&lt;br/&gt;
user    0m0.024s&lt;br/&gt;
sys     0m0.004s&lt;/p&gt;



&lt;p&gt;but it can connect to second node from same host:&lt;/p&gt;

&lt;p&gt;root@elba-mongo1:~#  time mongo --eval &quot;propmt=&apos;&amp;gt;&apos;&quot; --host elba-mongo2&lt;br/&gt;
MongoDB shell version: 2.2.2&lt;br/&gt;
connecting to: elba-mongo2:27017/test&lt;br/&gt;
&amp;gt;&lt;/p&gt;

&lt;p&gt;real    0m0.028s&lt;br/&gt;
user    0m0.020s&lt;br/&gt;
sys     0m0.008s&lt;/p&gt;


&lt;p&gt;At the same time, if I leave mongo client at elba-mongo2 connected to elba-mongo1, it can communicate with server, while new clients from other nodes (including localhost) can&apos;t connect. Here&apos;s rs.status() from elba-mongo1 at that moment:&lt;/p&gt;

&lt;p&gt;rs0:PRIMARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
        &quot;set&quot; : &quot;rs0&quot;,&lt;br/&gt;
        &quot;date&quot; : ISODate(&quot;2013-01-21T09:58:29Z&quot;),&lt;br/&gt;
        &quot;myState&quot; : 1,&lt;br/&gt;
        &quot;members&quot; : [&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 0,
                        &quot;name&quot; : &quot;elba-mongo1:27017&quot;,
                        &quot;health&quot; : 1,
                        &quot;state&quot; : 1,
                        &quot;stateStr&quot; : &quot;PRIMARY&quot;,
                        &quot;uptime&quot; : 355478,
                        &quot;optime&quot; : Timestamp(1358762294000, 4),
                        &quot;optimeDate&quot; : ISODate(&quot;2013-01-21T09:58:14Z&quot;),
                        &quot;self&quot; : true
                }
&lt;p&gt;,&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 1,
                        &quot;name&quot; : &quot;elba-mongo2:27017&quot;,
                        &quot;health&quot; : 1,
                        &quot;state&quot; : 2,
                        &quot;stateStr&quot; : &quot;SECONDARY&quot;,
                        &quot;uptime&quot; : 478,
                        &quot;optime&quot; : Timestamp(1358754311000, 7),
                        &quot;optimeDate&quot; : ISODate(&quot;2013-01-21T07:45:11Z&quot;),
                        &quot;lastHeartbeat&quot; : ISODate(&quot;2013-01-21T09:58:27Z&quot;),
                        &quot;pingMs&quot; : 300
                }
&lt;p&gt;,&lt;/p&gt;
                {
                        &quot;_id&quot; : 2,
                        &quot;name&quot; : &quot;elba-mongo3:27017&quot;,
                        &quot;health&quot; : 1,
                        &quot;state&quot; : 1,
                        &quot;stateStr&quot; : &quot;PRIMARY&quot;,
                        &quot;uptime&quot; : 9145,
                        &quot;optime&quot; : Timestamp(1358762308000, 7),
                        &quot;optimeDate&quot; : ISODate(&quot;2013-01-21T09:58:28Z&quot;),
                        &quot;lastHeartbeat&quot; : ISODate(&quot;2013-01-21T09:58:29Z&quot;),
                        &quot;pingMs&quot; : 0
                }
&lt;p&gt;        ],&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;



&lt;p&gt;I noticed, that mongod process is often in uninterruptable sleep state: &lt;br/&gt;
mongodb  17568  0.5 35.1 161820972 17377724 ?  Dl   Jan17  33:28 /usr/bin/mongod --config /etc/mongodb.conf&lt;/p&gt;

&lt;p&gt;I&apos;m attaching strace output for mongod from elba-mongo1. It was started slightly before last heartbeat from elba-mongo1 received by elba-mongo2 and lasts until mongo client connets to elba-mongo1.&lt;/p&gt;

&lt;p&gt;I&apos;m also seeing from logs, that mongod is running with NUMA enabled. Can it be casue for this issue?&lt;/p&gt;

</comment>
                            <comment id="242769" author="tad" created="Thu, 17 Jan 2013 18:42:33 +0000"  >&lt;p&gt;Hi Renat,&lt;/p&gt;

&lt;p&gt;The cases where a replica set has two primaries are caused by a network partition, particularly an asymmetrical partition where node 1 can see the node 2 but node 2 cannot see node 1.  As you saw, these conditions are temporary, because eventually the original primary will see that there is no longer a majority and will step down.&lt;/p&gt;

&lt;p&gt;It may require more exhaustive tests of your network than you have performed so far to narrow down the problem.  Can every node ping every other node continuously while you are seeing these problems?  Are the pings equally reliable with large packets? &lt;/p&gt;

&lt;p&gt;We have been able to reproduce issues similar to yours by simulating delays on the network that exceed the heartbeat frequency and we are looking into ways to increase reliability in the presence of network problems.&lt;/p&gt;

&lt;p&gt;But we are unable to see these issues on a fully functional network.  Your logs show replication cycles that begin, proceed for a while and then fail with socket errors, and do this over and over again.  It seems that something is wrong at the network layer; this could be a configuration problem or an intermittent segment, router or switch, a network card or interface that sometimes corrupts data or almost anything that prevents packets from being sent and received error-free between all nodes in a timely manner.&lt;/p&gt;

&lt;p&gt;Tad&lt;/p&gt;</comment>
                            <comment id="242655" author="renadeen" created="Thu, 17 Jan 2013 16:44:55 +0000"  >&lt;p&gt;(I&apos;m Petr&apos;s co-worker) Today we observed another issue. We had elba-mongo1 as PRIMARY and elba-mongo2 and elba-mongo3 as SECONDARY. Suddenly second and third stopped to see first instance (i.e. rs.status() outputs that it is not reachable/healthy). But first continued to send and receive heartbeats to second and third and preserved its PRIMARY state! After a while second became primary and replica set began to accept client requests. This situation (elba-mongo1 is seeing others but others isn&apos;t see it and there is two primaries) continued for several minutes. There were some another clutter with elections which you can see on logs (mongo1.log, etc.). It continued until I restared mongo instance on elba-mongo1. That is it is unlikely that it is network problem. Probably it is issue of listener of heartbeats.&lt;br/&gt;
How can it be that there were two primaries and elba-mongo1 didn&apos;t step down when it had seen another primary? I&apos;ve attached logs from all machines and rs.status() output during the disaster.&lt;/p&gt;</comment>
                            <comment id="241537" author="gusev_p" created="Wed, 16 Jan 2013 16:07:40 +0000"  >&lt;p&gt;Hi Tad&lt;/p&gt;

&lt;p&gt;1)I will try your suggestion about mongo shell next time we experience the problem. Usually this happen under daily load.&lt;br/&gt;
2)Can we definitly determine is this a networking problem or server side node listening problem ? We run some networking tests and they report ok.&lt;/p&gt;</comment>
                            <comment id="241506" author="tad" created="Wed, 16 Jan 2013 15:41:28 +0000"  >&lt;p&gt;Hi Gusev,&lt;/p&gt;

&lt;p&gt;You log file shows a number of rollback attempts, some of which succeed but most of which fail.  It looks like there were network problems during this period; several like &quot;exception: 10278 dbclient error communicating with server: elba-mongo2:27017&quot; and like &quot;replSet info elba-mongo2:27017 is down (or slow to respond): socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;SEND_ERROR&amp;#93;&lt;/span&gt; for 192.168.116.150:27017&quot;.&lt;/p&gt;

&lt;p&gt;The mongo shell attempts some communication with the server to determine the prompt to display (e.g. &quot;PRIMARY&amp;gt; &quot; or &quot;SECONDARY&amp;gt; &quot;) and this may be where the shell is appearing to hang.&lt;/p&gt;

&lt;p&gt;Can you try running the shell with a command line to disable this prompt calculation and see if this enables you to connect?  Try this command at the bash prompt:&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;mongo --eval &quot;prompt=&apos;&amp;gt;&amp;gt;&amp;gt; &apos; &quot; --shell&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This will force the prompt to be &quot;&amp;gt;&amp;gt;&amp;gt; &quot; (you can substitute anything you want), which will enable you to see if you can connect to the server even if the prompt negotiation is not working for some reason due to the state of the server.  (The &quot;&amp;#45;&amp;#45;eval&quot; part passes a JavaScript command to the shell, &quot;prompt=&apos;text&apos; &quot; sets the &quot;prompt&quot; variable and &quot;&amp;#45;&amp;#45;shell&quot; tells the shell not to exit after executing the JavaScript).&lt;/p&gt;

&lt;p&gt;Tad&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="48508">SERVER-6889</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="23190" name="log" size="89286" author="gusev_p" created="Wed, 16 Jan 2013 14:16:53 +0000"/>
                            <attachment id="23232" name="mongo1.log" size="45718" author="renadeen" created="Thu, 17 Jan 2013 16:45:55 +0000"/>
                            <attachment id="23233" name="mongo2.log" size="136940" author="renadeen" created="Thu, 17 Jan 2013 16:45:55 +0000"/>
                            <attachment id="23234" name="mongo3.log" size="116945" author="renadeen" created="Thu, 17 Jan 2013 16:45:55 +0000"/>
                            <attachment id="23231" name="rsStatus.txt" size="5803" author="renadeen" created="Thu, 17 Jan 2013 16:45:55 +0000"/>
                            <attachment id="23307" name="strace.out" size="140901" author="demonixed" created="Mon, 21 Jan 2013 11:27:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 16 Jan 2013 15:41:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 4 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 4 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>demonixed</customfieldvalue>
            <customfieldvalue>gusev_p</customfieldvalue>
            <customfieldvalue>renadeen</customfieldvalue>
            <customfieldvalue>tad</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnbi7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrlmkf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>39702</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10166" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Tests Written</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10153"><![CDATA[Unneeded]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsq0cn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>