<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:11:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-6300] Node 2 is master but attempts to replicate from Node 3</title>
                <link>https://jira.mongodb.org/browse/SERVER-6300</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I have been running some replication tests, and found something quite puzzling regarding failovers.&lt;/p&gt;

&lt;p&gt;Looking at packet capture off the bridge interface (attached) and log files (snippets attached): After failure on network connection of Node 1 (which was master, last successful communications around packet 2414), Node 2 became master (2566 seems to be the first heartbeat with Node 2 as master). The client eventually timed out as expected. When the client attempted to perform the next update on Node 2 (packets 3271 and 3273), the write timed out. It appears that Node 2 thinks it is master, but instead of Node 3 replicating from Node 2, Node 2 is instead asking for replication updated from Node 3 (e.g. 3288). I think this is a bug.&lt;/p&gt;

&lt;p&gt;The replication setup is a standard 3-node replica set. The nodes run in separate VirtualBox VMs with bridged networking. The network interfaces are connected to TAP interfaces on the host and connected together using a bridge device. In the test in question, a single client running on the host updates numbers in collection &apos;test&apos; on the cluster one at a time. A failure is introduced by killing network connection of one of the VMs by setting packet loss on its TAP interface to 100% with netem.&lt;/p&gt;

&lt;p&gt;Snippet from mongodb.log of Node 2 when it became master:&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; DBClientCursor::init call() failed&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info test1:27017 is down (or slow to respond): DBClientBase::findN: transport error: test1:27017 query: &lt;/p&gt;
{ replSetHeartbeat: &quot;test&quot;, v: 1, pv: 1, checkEmpty: false, from: &quot;test2:27017&quot; }
&lt;p&gt;Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test1:27017 is now in state DOWN&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info electSelf 2&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet PRIMARY&lt;/p&gt;

&lt;p&gt;Snippet from mongodb.log of Node 3 when it formed new cluster with Node 2:&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn190&amp;#93;&lt;/span&gt; replSet info voting yea for test2:27017 (2)&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; couldn&apos;t connect to test1:27017: couldn&apos;t connect to server test1:27017&lt;br/&gt;
Wed Jul  4 16:44:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test2:27017 is now in state PRIMARY&lt;br/&gt;
Wed Jul  4 16:44:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info two primaries (transiently)&lt;br/&gt;
Wed Jul  4 16:44:44 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info two primaries (transiently)&lt;br/&gt;
Wed Jul  4 16:44:47 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info test1:27017 is down (or slow to respond): socket exception&lt;br/&gt;
Wed Jul  4 16:44:47 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test1:27017 is now in state DOWN&lt;/p&gt;</description>
                <environment></environment>
        <key id="43368">SERVER-6300</key>
            <summary>Node 2 is master but attempts to replicate from Node 3</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="tuure.laurinolli@portalify.com">Tuure Laurinolli</reporter>
                        <labels>
                    </labels>
                <created>Wed, 4 Jul 2012 15:13:42 +0000</created>
                <updated>Mon, 11 Jul 2016 18:34:15 +0000</updated>
                            <resolved>Wed, 29 Aug 2012 19:13:47 +0000</resolved>
                                    <version>2.0.6</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="158169" author="kristina" created="Wed, 29 Aug 2012 19:13:47 +0000"  >&lt;p&gt;Please leave a comment if you have other concerns, otherwise I&apos;ll assume discussion will move to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6733&quot; title=&quot;Make oplog timeout shorter&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6733&quot;&gt;&lt;del&gt;SERVER-6733&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="151603" author="kristina" created="Wed, 8 Aug 2012 15:18:29 +0000"  >&lt;p&gt;The timeout was raised because it was causing spurious timeouts in tests.  It was a &quot;magic number&quot; choice, though.&lt;/p&gt;

&lt;p&gt;I see what you mean about &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5208&quot; title=&quot;Replica set periodic reevaluation of sync targets&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5208&quot;&gt;&lt;del&gt;SERVER-5208&lt;/del&gt;&lt;/a&gt;.  Lowering the timeout might be worth revisiting post-2.2, as there have been substantial changes in the replication code that should make shorter timeouts work better.  I created a ticket for this: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6733&quot; title=&quot;Make oplog timeout shorter&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6733&quot;&gt;&lt;del&gt;SERVER-6733&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="149472" author="tuure.laurinolli@portalify.com" created="Wed, 1 Aug 2012 14:37:10 +0000"  >&lt;p&gt;Revisiting this now that I got back to MongoDB in my testing - the default timeout of 10 minutes seems rather high. Contrast with the timeout of 20 seconds set as solution to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4918&quot; title=&quot;Lower replica set reader timeout (or make it configurable)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4918&quot;&gt;&lt;del&gt;SERVER-4918&lt;/del&gt;&lt;/a&gt;. Is there actually some rationale behind having the timeout be in the order of minutes instead of seconds?&lt;/p&gt;

&lt;p&gt;The commits in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4758&quot; title=&quot;OplogReader has no socket timeout&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4758&quot;&gt;&lt;del&gt;SERVER-4758&lt;/del&gt;&lt;/a&gt; mention raising it from 60 seconds to 10 minutes so presumably there are some bad experiences behind the change. As it is, having the cluster be unwritable (with w:2) for 10 minutes after a network partition seems rather silly. I&apos;m not sure how &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5208&quot; title=&quot;Replica set periodic reevaluation of sync targets&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5208&quot;&gt;&lt;del&gt;SERVER-5208&lt;/del&gt;&lt;/a&gt; would help either, unless it would somehow unblock the replication connection.&lt;/p&gt;</comment>
                            <comment id="143294" author="kristina" created="Mon, 16 Jul 2012 17:24:23 +0000"  >&lt;p&gt;You could file a feature request to make it tunable, but it&apos;s unlikely to be accepted because MongoDB&apos;s general philosophy on the subject is &quot;no knobs&quot; (so it&apos;s difficult get a new one).  &lt;/p&gt;

&lt;p&gt;It seems reasonable to me that changes in the set would cause a re-evaluation of replication target, I added a comment about it to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-5208&quot; title=&quot;Replica set periodic reevaluation of sync targets&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-5208&quot;&gt;&lt;del&gt;SERVER-5208&lt;/del&gt;&lt;/a&gt;, which seems like the relevant issue.&lt;/p&gt;</comment>
                            <comment id="143266" author="tuure.laurinolli@portalify.com" created="Mon, 16 Jul 2012 15:55:43 +0000"  >&lt;p&gt;Could the timeout be made tunable? Or better yet, could change in cluster configuration interrupt the blocking replication reads, causing them to take updated cluster configuration into account?&lt;/p&gt;</comment>
                            <comment id="142909" author="kristina" created="Fri, 13 Jul 2012 21:16:03 +0000"  >&lt;p&gt;Unfortunately no, unless you recompile :-/&lt;/p&gt;</comment>
                            <comment id="142899" author="tuure.laurinolli@portalify.com" created="Fri, 13 Jul 2012 20:53:56 +0000"  >&lt;p&gt;At 16:47 I brought Node 1 back up. Is the timeout tunable? I would like something in the order of seconds.&lt;/p&gt;</comment>
                            <comment id="142878" author="kristina" created="Fri, 13 Jul 2012 19:30:28 +0000"  >&lt;p&gt;Hmm, looks like we added a timeout to the replication thread in 2.0.6 (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4758&quot; title=&quot;OplogReader has no socket timeout&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4758&quot;&gt;&lt;del&gt;SERVER-4758&lt;/del&gt;&lt;/a&gt;) that is set for 10 minutes.  Did it figure out who to sync from after that?&lt;/p&gt;</comment>
                            <comment id="142132" author="tuure.laurinolli@portalify.com" created="Wed, 11 Jul 2012 22:47:23 +0000"  >&lt;p&gt;It appears that Node 3 never started syncin from Node2:&lt;/p&gt;

&lt;p&gt;Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn190&amp;#93;&lt;/span&gt; replSet info voting yea for test2:27017 (2)&lt;br/&gt;
Wed Jul  4 16:44:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; couldn&apos;t connect to test1:27017: couldn&apos;t connect to server test1:27017&lt;br/&gt;
Wed Jul  4 16:44:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test2:27017 is now in state PRIMARY&lt;br/&gt;
Wed Jul  4 16:44:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info two primaries (transiently)&lt;br/&gt;
Wed Jul  4 16:44:44 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsMgr&amp;#93;&lt;/span&gt; replSet info two primaries (transiently)&lt;br/&gt;
Wed Jul  4 16:44:47 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet info test1:27017 is down (or slow to respond): socket exception&lt;br/&gt;
Wed Jul  4 16:44:47 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test1:27017 is now in state DOWN&lt;br/&gt;
Wed Jul  4 16:44:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn191&amp;#93;&lt;/span&gt; end connection 192.168.1.1:42628&lt;br/&gt;
Wed Jul  4 16:44:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn190&amp;#93;&lt;/span&gt; end connection 192.168.1.2:44767&lt;br/&gt;
Wed Jul  4 16:44:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.2:44771 #192&lt;br/&gt;
Wed Jul  4 16:45:12 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.100:35964 #193&lt;br/&gt;
Wed Jul  4 16:45:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn192&amp;#93;&lt;/span&gt; end connection 192.168.1.2:44771&lt;br/&gt;
Wed Jul  4 16:45:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.2:44774 #194&lt;br/&gt;
Wed Jul  4 16:45:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn194&amp;#93;&lt;/span&gt; end connection 192.168.1.2:44774&lt;br/&gt;
Wed Jul  4 16:45:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.2:44778 #195&lt;br/&gt;
Wed Jul  4 16:46:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn195&amp;#93;&lt;/span&gt; end connection 192.168.1.2:44778&lt;br/&gt;
Wed Jul  4 16:46:22 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.2:44781 #196&lt;br/&gt;
Wed Jul  4 16:46:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn196&amp;#93;&lt;/span&gt; end connection 192.168.1.2:44781&lt;br/&gt;
Wed Jul  4 16:46:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 192.168.1.2:44785 #197&lt;br/&gt;
Wed Jul  4 16:47:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test1:27017 is up&lt;br/&gt;
Wed Jul  4 16:47:00 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsHealthPoll&amp;#93;&lt;/span&gt; replSet member test1:27017 is now in state SECONDARY&lt;/p&gt;

&lt;p&gt;This is also clear from the packet capture. For example at 16:45:01 (packet 2894) Node 2 is still trying to replicate from Node 3. The packet capture also shows TCP retransmissions of Get More requests from Node 3 to Node 1 tens of seconds after Node 1 has stopped responding (e.g. packet 3139, 16:45:22). Perhaps the replication connections do not time out for some reason?&lt;/p&gt;</comment>
                            <comment id="142087" author="kristina" created="Wed, 11 Jul 2012 21:19:32 +0000"  >&lt;p&gt;&amp;gt; How long time should it take before the replication thread notices that it is primary? &lt;/p&gt;

&lt;p&gt;Generally less than a second.  The other members will usually take longer to notice (usually less than 2 seconds but up to 20 seconds in the worst case).&lt;/p&gt;

&lt;p&gt;&amp;gt; how long before the replication thread on Node 3 notices that it should be replicating &lt;br/&gt;
&amp;gt; from Node 2 (as it doesn&apos;t seem to be, if you look at the packet capture)?&lt;/p&gt;

&lt;p&gt;Once it realizes Node 1 is down and Node 2 is primary.  You should be able to see when Node 3 starts syncing by looking at its log.  It&apos;ll have the DOWN message you pasted above and then (probably a few lines below?) should have a &quot;syncing to: host:port&quot; line when it starts syncing from Node 2.&lt;/p&gt;

&lt;p&gt;During various failure scenarios, you may have w:2 timeout depending on how long your timeout is.&lt;/p&gt;</comment>
                            <comment id="141892" author="tuure.laurinolli@portalify.com" created="Wed, 11 Jul 2012 15:21:29 +0000"  >&lt;p&gt;How long time should it take before the replication thread notices that it is primary? And more importantly, how long before the replication thread on Node 3 notices that it should be replicating from Node 2 (as it doesn&apos;t seem to be, if you look at the packet capture)?&lt;/p&gt;

&lt;p&gt;The issue is that writes with w:2 time out because replication from Node 2 to Node 3 does not happen.&lt;/p&gt;</comment>
                            <comment id="141881" author="kristina" created="Wed, 11 Jul 2012 15:01:34 +0000"  >&lt;p&gt;This is by design. Replication is on a separate thread from heartbeats, so there may be some delay before the replication thread notices that it is primary.  However, the replication thread is careful to check that it is not primary before actually &lt;em&gt;applying&lt;/em&gt; ops, see &lt;a href=&quot;https://github.com/mongodb/mongo/blob/v2.0/db/repl/rs_sync.cpp#L427&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/v2.0/db/repl/rs_sync.cpp#L427&lt;/a&gt;.  (syncApply() is the function that actually applies the operation to the member.)&lt;/p&gt;

&lt;p&gt;Are you having an issue that you thought was caused by this or were you just concerned about the network traffic you observed?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="46583">SERVER-6733</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="17942" name="br0.pcap" size="691495" author="tuure.laurinolli@portalify.com" created="Wed, 4 Jul 2012 15:13:42 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 11 Jul 2012 15:01:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 25 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 25 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>tuure.laurinolli@portalify.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnxhz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrghtj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9825</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0ob3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>