<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:06:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4709] Strange timing of operations reported in logs</title>
                <link>https://jira.mongodb.org/browse/SERVER-4709</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While checking logs for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4100&quot; title=&quot;Please provide Windows mongodb client program in MSI format&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4100&quot;&gt;&lt;del&gt;SERVER-4100&lt;/del&gt;&lt;/a&gt; causes (this case is unrelated, mentioned for information purposes only), I got some strange reconds on the primary:&lt;/p&gt;

&lt;p&gt;Sat Jan 14 02:15:37 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.2.1.200:48055 #108356&lt;br/&gt;
Sat Jan 14 02:15:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn108356&amp;#93;&lt;/span&gt; update models_stats.show_log query: &lt;/p&gt;
{ _id: ObjectId(&apos;4ebf7eb76b8dda061480d539&apos;) }
&lt;p&gt; update: { $unset: &lt;/p&gt;
{ h: 1 }
&lt;p&gt;, $set: &lt;/p&gt;
{ e: null, s: 1321172663 }
&lt;p&gt; } keyUpdates:1 1271310315ms&lt;br/&gt;
Sat Jan 14 02:15:53 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn108356&amp;#93;&lt;/span&gt; end connection 10.2.1.200:48055&lt;/p&gt;

&lt;p&gt;This part of log shows the update running for something around 14 days, but entire connection existed less than a minute, looks like there&apos;s something amiss.&lt;br/&gt;
We have several (in the tens range) of such queries in logs with the same time (1271310315ms) on different connections with different operations.&lt;/p&gt;</description>
                <environment></environment>
        <key id="28568">SERVER-4709</key>
            <summary>Strange timing of operations reported in logs</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="benjamin.becker">Ben Becker</assignee>
                                    <reporter username="onyxmaster">Aristarkh Zagorodnikov</reporter>
                        <labels>
                    </labels>
                <created>Wed, 18 Jan 2012 09:14:11 +0000</created>
                <updated>Wed, 29 Feb 2012 03:54:00 +0000</updated>
                            <resolved>Wed, 25 Jan 2012 18:25:34 +0000</resolved>
                                    <version>2.0.2</version>
                                                    <component>Logging</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="82328" author="benjamin.becker" created="Wed, 25 Jan 2012 18:24:54 +0000"  >&lt;p&gt;I&apos;m going to close this issue for now.  Please reopen if you encounter this issue again.&lt;/p&gt;</comment>
                            <comment id="81708" author="benjamin.becker" created="Mon, 23 Jan 2012 18:34:39 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4740&quot; title=&quot;Use monotonic clock sources for Timer&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-4740&quot;&gt;&lt;del&gt;SERVER-4740&lt;/del&gt;&lt;/a&gt; may help with strange time results.&lt;/p&gt;</comment>
                            <comment id="81510" author="onyxmaster" created="Sun, 22 Jan 2012 08:50:36 +0000"  >&lt;p&gt;I tend to blame virtualization, because we ran the same workload on physical hardware, I checked logs and there was no similar problems.&lt;br/&gt;
I also would like to not that this effect was demonstrated twice, when system was under very high (unusually, to the point of can&apos;t-type-in-SSH) load.&lt;/p&gt;</comment>
                            <comment id="81506" author="onyxmaster" created="Sun, 22 Jan 2012 07:27:27 +0000"  >&lt;p&gt;Will do next week.&lt;/p&gt;</comment>
                            <comment id="81497" author="benjamin.becker" created="Sun, 22 Jan 2012 04:08:50 +0000"  >&lt;p&gt;It&apos;s worth noting that mongo does not use a monotonic timer for most stats, and thus any change in system time will affect the results.  This can happen due to a Xen live migration, system suspend, NTP time change, user-initiated time change, etc...  &lt;/p&gt;

&lt;p&gt;If this issue is on-going (and you have the time), could you try building the attached test.c file and let it run while the issue appears?  It runs &lt;em&gt;gettimeofday()&lt;/em&gt; and &lt;em&gt;clock_gettime()&lt;/em&gt; every ~50ms, and shouldn&apos;t use too much of the CPU.  Please post back any output from this program, or if you observe another invalid time while the program does not report anything.&lt;/p&gt;

&lt;p&gt;To build and run (assuming test.c is in the current directory):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;gcc ./test -otest -lrt &amp;amp;&amp;amp; ./test&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks for the report; I&apos;ll see if we can move towards safer, monotonic timing functions in the future.&lt;/p&gt;</comment>
                            <comment id="81486" author="onyxmaster" created="Sat, 21 Jan 2012 22:40:40 +0000"  >&lt;p&gt;This is a custom built Xen kernel, and no source is available currently since we don&apos;t control the environment (we lease the environment, think small Amazon-kind cloud that sells servcies for us, we even don&apos;t know what kind of hardware runs this). According to this we can&apos;t force them to update the kernel.&lt;br/&gt;
Currently, the issue doesn&apos;t give us any problems, I just posted to inform you about the log inconsistencies.&lt;/p&gt;

&lt;p&gt;I would check the logs as you asked, but don&apos;t hold your breath.&lt;/p&gt;</comment>
                            <comment id="81477" author="benjamin.becker" created="Sat, 21 Jan 2012 21:02:16 +0000"  >&lt;p&gt;Hm, I haven&apos;t found anything in the mongo source code that could be causing the strange time values (yet).  This sounds like a bug in the kernel or hypervisor.  Is the source and config for this kernel (v2.6.32.36-228-scalaxy) available?  Is this OS running under KVM, Xen or another hypervisor?&lt;/p&gt;

&lt;p&gt;There have been several patches related to invalid time values in the linux-stable 2.6.32.y branch since 2011-04-14, and both KVM and Xen have had significant TSC-related bugs in the past (some examples: &lt;a href=&quot;https://bugs.launchpad.net/ubuntu/+source/linux/+bug/727459&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Ubuntu on Xen bug&lt;/a&gt;, &lt;a href=&quot;http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=28e4639adf0c9f26f6bb56149b7ab547bf33bb95&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;KVM kvmclock() jump bug&lt;/a&gt;, &lt;a href=&quot;http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=1d5f066e0b63271b67eac6d3752f8aa96adcbddb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;KVM kvmclock() backwards bug&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Because the bogus log entries are transient, I would suspect an issue related to reading the TSC from another processor or vcpu.  Can you check the kernel log (&lt;em&gt;dmesg&lt;/em&gt;) for anything related to HPET, TSC, CLOCK, or skewing?  Do you have access to the hypervisor/VMM/Dom0 host?  There may also be some useful information in /proc/timer_list and /proc/cpuinfo.  Also, please note that certain processors (e.g. AMD K8 Rev.A-E) have known issues with APIC timers, HPET and/or TSC values.&lt;/p&gt;

&lt;p&gt;I&apos;ve had a test running for about 8 hours now in a virtualized Ubuntu 10.04 box (v2.6.32-36 on Parallels) that constantly runs curTimeMicros64() with no anomalies so far.  Is there any additional information you could provide that might help to reproduce this issue?&lt;/p&gt;</comment>
                            <comment id="81444" author="onyxmaster" created="Sat, 21 Jan 2012 09:48:49 +0000"  >&lt;p&gt;xm@dbs1-d:~$ uname -a&lt;br/&gt;
Linux dbs1-d.dc2.bru 2.6.32.36-228-scalaxy #1 SMP 2011-04-14 10:12:31 +0200 x86_64 GNU/Linux&lt;/p&gt;

&lt;p&gt;xm@dbs1-d:~$ mongo&lt;br/&gt;
MongoDB shell version: 2.0.2&lt;br/&gt;
connecting to: test&lt;br/&gt;
PRIMARY&amp;gt; db.serverStatus()&lt;br/&gt;
{&lt;br/&gt;
        &quot;host&quot; : &quot;dbs1-d.dc2.bru&quot;,&lt;br/&gt;
        &quot;version&quot; : &quot;2.0.2&quot;,&lt;br/&gt;
        &quot;process&quot; : &quot;mongod&quot;,&lt;br/&gt;
        &quot;uptime&quot; : 320149,&lt;br/&gt;
        &quot;uptimeEstimate&quot; : 266481,&lt;br/&gt;
        &quot;localTime&quot; : ISODate(&quot;2012-01-21T09:49:14.719Z&quot;),&lt;br/&gt;
        &quot;globalLock&quot; : {&lt;br/&gt;
                &quot;totalTime&quot; : 320149677975,&lt;br/&gt;
                &quot;lockTime&quot; : 417786069,&lt;br/&gt;
                &quot;ratio&quot; : 0.001304971073663314,&lt;br/&gt;
                &quot;currentQueue&quot; : &lt;/p&gt;
{
                        &quot;total&quot; : 0,
                        &quot;readers&quot; : 0,
                        &quot;writers&quot; : 0
                }
&lt;p&gt;,&lt;br/&gt;
                &quot;activeClients&quot; : &lt;/p&gt;
{
                        &quot;total&quot; : 1,
                        &quot;readers&quot; : 1,
                        &quot;writers&quot; : 0
                }
&lt;p&gt;        },&lt;br/&gt;
        &quot;mem&quot; : &lt;/p&gt;
{
                &quot;bits&quot; : 64,
                &quot;resident&quot; : 1992,
                &quot;virtual&quot; : 9925,
                &quot;supported&quot; : true,
                &quot;mapped&quot; : 4670,
                &quot;mappedWithJournal&quot; : 9340
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;connections&quot; : &lt;/p&gt;
{
                &quot;current&quot; : 9,
                &quot;available&quot; : 15991
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;extra_info&quot; : &lt;/p&gt;
{
                &quot;note&quot; : &quot;fields vary by platform&quot;,
                &quot;heap_usage_bytes&quot; : 940880,
                &quot;page_faults&quot; : 1698
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;indexCounters&quot; : {&lt;br/&gt;
                &quot;btree&quot; : &lt;/p&gt;
{
                        &quot;accesses&quot; : 70308,
                        &quot;hits&quot; : 70308,
                        &quot;misses&quot; : 0,
                        &quot;resets&quot; : 0,
                        &quot;missRatio&quot; : 0
                }
&lt;p&gt;        },&lt;br/&gt;
        &quot;backgroundFlushing&quot; : &lt;/p&gt;
{
                &quot;flushes&quot; : 5330,
                &quot;total_ms&quot; : 468517,
                &quot;average_ms&quot; : 87.90187617260788,
                &quot;last_ms&quot; : 8,
                &quot;last_finished&quot; : ISODate(&quot;2012-01-21T09:48:55.927Z&quot;)
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;cursors&quot; : &lt;/p&gt;
{
                &quot;totalOpen&quot; : 1,
                &quot;clientCursors_size&quot; : 1,
                &quot;timedOut&quot; : 0
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;network&quot; : &lt;/p&gt;
{
                &quot;bytesIn&quot; : 239991766,
                &quot;bytesOut&quot; : 4429271600,
                &quot;numRequests&quot; : 3389982
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;repl&quot; : &lt;/p&gt;
{
                &quot;setName&quot; : &quot;stat.drive.ru&quot;,
                &quot;ismaster&quot; : true,
                &quot;secondary&quot; : false,
                &quot;hosts&quot; : [
                        &quot;dbs1-d.dc2.bru:27017&quot;,
                        &quot;dbs2-d.dc2.bru:27017&quot;
                ],
                &quot;arbiters&quot; : [
                        &quot;dbsa-d.dc2.bru:27023&quot;
                ],
                &quot;primary&quot; : &quot;dbs1-d.dc2.bru:27017&quot;,
                &quot;me&quot; : &quot;dbs1-d.dc2.bru:27017&quot;
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;opcounters&quot; : &lt;/p&gt;
{
                &quot;insert&quot; : 1001856,
                &quot;query&quot; : 92647,
                &quot;update&quot; : 83252,
                &quot;delete&quot; : 0,
                &quot;getmore&quot; : 1087289,
                &quot;command&quot; : 1124967
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;asserts&quot; : &lt;/p&gt;
{
                &quot;regular&quot; : 0,
                &quot;warning&quot; : 0,
                &quot;msg&quot; : 0,
                &quot;user&quot; : 3,
                &quot;rollovers&quot; : 0
        }
&lt;p&gt;,&lt;br/&gt;
        &quot;writeBacksQueued&quot; : false,&lt;br/&gt;
        &quot;dur&quot; : {&lt;br/&gt;
                &quot;commits&quot; : 84,&lt;br/&gt;
                &quot;journaledMB&quot; : 0.114688,&lt;br/&gt;
                &quot;writeToDataFilesMB&quot; : 0.02319,&lt;br/&gt;
                &quot;compression&quot; : 0.9999912807679899,&lt;br/&gt;
                &quot;commitsInWriteLock&quot; : 0,&lt;br/&gt;
                &quot;earlyCommits&quot; : 0,&lt;br/&gt;
                &quot;timeMs&quot; : &lt;/p&gt;
{
                        &quot;dt&quot; : 3032,
                        &quot;prepLogBuffer&quot; : 0,
                        &quot;writeToJournal&quot; : 8,
                        &quot;writeToDataFiles&quot; : 0,
                        &quot;remapPrivateView&quot; : 0
                }
&lt;p&gt;        },&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;br/&gt;
PRIMARY&amp;gt; rs.config()&lt;br/&gt;
{&lt;br/&gt;
        &quot;_id&quot; : &quot;stat.drive.ru&quot;,&lt;br/&gt;
        &quot;version&quot; : 2,&lt;br/&gt;
        &quot;members&quot; : [&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 0,
                        &quot;host&quot; : &quot;dbsa-d.dc2.bru:27023&quot;,
                        &quot;arbiterOnly&quot; : true
                }
&lt;p&gt;,&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 1,
                        &quot;host&quot; : &quot;dbs1-d.dc2.bru:27017&quot;,
                        &quot;priority&quot; : 10
                }
&lt;p&gt;,&lt;/p&gt;
                {
                        &quot;_id&quot; : 2,
                        &quot;host&quot; : &quot;dbs2-d.dc2.bru:27017&quot;
                }
&lt;p&gt;        ]&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;Nothing specific (alarming) found in logs, plain connects/disconnects, occasional 100-200ms queries.&lt;/p&gt;</comment>
                            <comment id="81442" author="benjamin.becker" created="Sat, 21 Jan 2012 06:17:17 +0000"  >&lt;p&gt;Could you provide the output of the shell command &lt;em&gt;uname -a&lt;/em&gt; and the output of the mongo command &lt;em&gt;db.serverStatus()&lt;/em&gt;?  Also, is this a single-server deployment or are there replica sets and/or multiple shards?  Is there any additional log output near what was initially provided?&lt;/p&gt;</comment>
                            <comment id="80798" author="onyxmaster" created="Wed, 18 Jan 2012 21:40:14 +0000"  >&lt;p&gt;xm@dbs1-d:~$ mongod --version&lt;br/&gt;
db version v2.0.2, pdfile version 4.5&lt;br/&gt;
Thu Jan 19 01:39:50 git version: 514b122d308928517f5841888ceaa4246a7f18e3&lt;/p&gt;</comment>
                            <comment id="80745" author="eliot" created="Wed, 18 Jan 2012 17:09:44 +0000"  >&lt;p&gt;Are you sure this is 2.0.2?&lt;br/&gt;
There was a bug &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3910&quot; title=&quot;millis always 0 when profilingLevel is 2&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3910&quot;&gt;&lt;del&gt;SERVER-3910&lt;/del&gt;&lt;/a&gt; in 2.0.0 that was fixed in 2.0.1&lt;/p&gt;</comment>
                            <comment id="80663" author="onyxmaster" created="Wed, 18 Jan 2012 09:15:30 +0000"  >&lt;p&gt;All of them are either insert or update operations, no queries.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="28815">SERVER-4740</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="14509" name="test.c" size="1907" author="benjamin.becker" created="Sun, 22 Jan 2012 04:09:01 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 18 Jan 2012 17:09:44 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 4 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 4 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>onyxmaster</customfieldvalue>
            <customfieldvalue>benjamin.becker</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrogkn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriton:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23462</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0ck7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>