<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:53:56 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40065] spike latency issue on shard cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-40065</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;on MongoDB 3.4.19, shard cluster with 2 shards.&lt;/p&gt;

&lt;p&gt;collection `foo` is sharded with&#160; hashed key &quot;UID&quot;&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&amp;gt; sh.enableSharding(&quot;foo&quot;)&lt;br/&gt;
&amp;gt; db.adminCommand({&quot;shardCollection&quot;: &quot;foo.foo&quot;, key: {&quot;UID&quot;: &quot;hashed&quot;}})&lt;br/&gt;
&amp;gt; use foo&lt;br/&gt;
&amp;gt; db.foo.createIndex({&quot;name&quot;: 1})&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;we write a test case in mongo-cxx-driver to test the latency.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;auto nStart = std::chrono::high_resolution_clock::now();&lt;br/&gt;
 oDB&lt;span class=&quot;error&quot;&gt;&amp;#91;DB_TABLE_NAME&amp;#93;&lt;/span&gt;.update_one(document{}&amp;lt;&amp;lt;&quot;UID&quot;&amp;lt;&amp;lt; nUID &amp;lt;&amp;lt;finalize, oValue.view(), opt);&lt;br/&gt;
 auto nEnd = std::chrono::high_resolution_clock::now();&lt;br/&gt;
 std::chrono::duration&amp;lt;double, std::milli&amp;gt; diff = nEnd - nStart;&lt;br/&gt;
 printf(&quot;uid %d timediff %f\n&quot;, nUID, diff.count());&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;output as below:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;uid 10030116 timediff 1.249126&lt;br/&gt;
uid 10000124 timediff 1.021864&lt;br/&gt;
uid 10050127 timediff 1.172118&lt;br/&gt;
uid 10020116 timediff 1.223791&lt;br/&gt;
uid 10040115 timediff 1.408828&lt;br/&gt;
uid 10070114 timediff 1.526046&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;What we did:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The test case run with 8 threads, each thread runs 10000 upserts loop to the collection.&lt;/li&gt;
	&lt;li&gt;Each upsert operation is on different UID.&lt;/li&gt;
	&lt;li&gt;Except UID/ID, all docs are always the same.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;What we found:&lt;/p&gt;

&lt;p&gt;1. the latency has many spike, such as 4000+ ms in some update_one operation, but not any slow log in mongod server(we set slowms to 10ms).&lt;br/&gt;
2. as 1, the total timediff is unstable(FROM: ./testmongo | awk &apos;{s+=$NF}END{print s}&apos;)&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;149726&lt;br/&gt;
129175&lt;br/&gt;
124993&lt;br/&gt;
219767&lt;br/&gt;
137422&lt;br/&gt;
156674&lt;br/&gt;
162410&lt;br/&gt;
119684&lt;br/&gt;
117086&lt;br/&gt;
116885&lt;/p&gt;&lt;/blockquote&gt;


&lt;p&gt;3. after we run `rs.stepDown()` on config Primary, the spike gone away(FROM: ./testmongo | awk &apos;{s+=$NF}END{print s}&apos;)&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;106808&lt;br/&gt;
107225&lt;br/&gt;
107228&lt;br/&gt;
108055&lt;br/&gt;
106660&lt;br/&gt;
108690&lt;br/&gt;
105993&lt;br/&gt;
107037&lt;br/&gt;
106226&lt;br/&gt;
104789&lt;br/&gt;
106494&lt;br/&gt;
105178&lt;br/&gt;
107428&lt;br/&gt;
108650&lt;br/&gt;
106535&lt;/p&gt;

&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;What we tried:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;initial the test db without sharding but create index UID as hashed, name as 1, the latency is normal and stable;&lt;/li&gt;
	&lt;li&gt;run the benchmark to each replicaSet, teh latency is normal and stable;&lt;/li&gt;
	&lt;li&gt;try to stepDown replicaSet, the latency may return normal.(not always)&lt;/li&gt;
	&lt;li&gt;try to stepDown config again, the latecy may abnormal/normal again.(not always)&lt;/li&gt;
	&lt;li&gt;replace the wiredTiger storage to MMAPv1, have the same sistuation.(can return normal after stepDown something)&lt;/li&gt;
	&lt;li&gt;try to stop balance, it has not help;&lt;/li&gt;
	&lt;li&gt;downgrade to MongoDB 3.2.22, the latency is normal and stable&lt;/li&gt;
	&lt;li&gt;not any abnormal log, we only found split point lookup and network connection logs.&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="714498">SERVER-40065</key>
            <summary>spike latency issue on shard cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="eric.sedor@mongodb.com">Eric Sedor</assignee>
                                    <reporter username="stutiredboy@gmail.com">Adun</reporter>
                        <labels>
                    </labels>
                <created>Mon, 11 Mar 2019 08:27:13 +0000</created>
                <updated>Mon, 22 Apr 2019 20:49:38 +0000</updated>
                            <resolved>Mon, 22 Apr 2019 20:40:04 +0000</resolved>
                                                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2221112" author="eric.sedor" created="Mon, 22 Apr 2019 20:38:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stutiredboy%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stutiredboy@gmail.com&quot;&gt;stutiredboy@gmail.com&lt;/a&gt;; we are closing this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40707&quot; title=&quot;Secondary couldn&amp;#39;t signal OplogWaiters to advance  the lastCommittedOpTime in Chained replication mode&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40707&quot;&gt;&lt;del&gt;SERVER-40707&lt;/del&gt;&lt;/a&gt; and encourage you to watch that ticket for updates.&lt;/p&gt;</comment>
                            <comment id="2217417" author="eric.sedor" created="Thu, 18 Apr 2019 04:26:23 +0000"  >&lt;p&gt;Thanks for pointing this out, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stutiredboy%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stutiredboy@gmail.com&quot;&gt;stutiredboy@gmail.com&lt;/a&gt;; we&apos;ll re-open this ticket, check on this relationship, and also see how we might have diagnosed this sooner.&lt;/p&gt;</comment>
                            <comment id="2217412" author="stutiredboy@gmail.com" created="Thu, 18 Apr 2019 03:56:15 +0000"  >&lt;p&gt;@Eric&lt;/p&gt;

&lt;p&gt;issue&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40707&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-40707&lt;/a&gt;&#160;is the root cause of this issue&lt;/p&gt;</comment>
                            <comment id="2185234" author="eric.sedor" created="Tue, 19 Mar 2019 19:32:01 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stutiredboy%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stutiredboy@gmail.com&quot;&gt;stutiredboy@gmail.com&lt;/a&gt;; unfortunately we&apos;re not seeing anything to indicate a bug. We would expect a server-side problem introducing this latency would result in logged slow ops. Without that, there are other possibilities such as network latency or a driver-side configuration issue such as connection pool size.&lt;/p&gt;

&lt;p&gt;As this ticket does not appear to a bug, I will now close it. If you need further assistance troubleshooting, please post on the &lt;a href=&quot;https://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or on &lt;a href=&quot;https://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;If you are able to narrow this experiment down to a reproducible bug, we&apos;d be interested in that.&lt;/p&gt;</comment>
                            <comment id="2184397" author="eric.sedor" created="Mon, 18 Mar 2019 21:28:09 +0000"  >&lt;p&gt;We appreciate your patience &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stutiredboy%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stutiredboy@gmail.com&quot;&gt;stutiredboy@gmail.com&lt;/a&gt;, and are continuing to review and discuss internally.&lt;/p&gt;</comment>
                            <comment id="2180581" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 09:36:44 +0000"  >&lt;p&gt;we run on MongoDB 3.4 again&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;$ ./mongotest_cpp34 | tee 1.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 107833&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 2log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 109542&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 3.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 109615&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 4.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 145013&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 5.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 126852&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 6.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 108585&lt;br/&gt;
 $ ./mongotest_cpp34 | tee 7.log | awk &apos;{s+=$NF}END{print s}&apos;&lt;br/&gt;
 149275&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;$ sort -k4 -n -r 7.log&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;uid 10044767 timediff 4279.467329&lt;br/&gt;
 uid 10031864 timediff 3968.625955&lt;br/&gt;
 uid 10021842 timediff 3941.699426&lt;br/&gt;
 uid 10011782 timediff 3871.421174&lt;br/&gt;
 uid 10033760 timediff 3579.863499&lt;br/&gt;
 uid 10052268 timediff 3087.380755&lt;br/&gt;
 uid 10062160 timediff 3071.985371&lt;br/&gt;
 uid 10013761 timediff 2878.962980&lt;br/&gt;
 uid 10075283 timediff 2830.154280&lt;br/&gt;
 uid 10023997 timediff 2735.049440&lt;br/&gt;
 uid 10064571 timediff 2127.151757&lt;br/&gt;
 uid 10002514 timediff 1921.079368&lt;br/&gt;
 uid 10004980 timediff 1776.391891&lt;br/&gt;
 uid 10060000 timediff 75.260478&lt;br/&gt;
 uid 10040000 timediff 73.680193&lt;br/&gt;
 uid 10030000 timediff 70.422951&lt;br/&gt;
 uid 10010000 timediff 69.084409&lt;br/&gt;
 uid 10020000 timediff 68.970113&lt;br/&gt;
 uid 10070000 timediff 68.486014&lt;br/&gt;
 uid 10050000 timediff 68.443622&lt;br/&gt;
 uid 10000000 timediff 64.418417&lt;br/&gt;
 uid 10012865 timediff 14.042705&lt;br/&gt;
 uid 10079510 timediff 13.110099&lt;/p&gt;

&lt;p&gt;......&lt;/p&gt;

&lt;p&gt;uid 10064410 timediff 0.944810&lt;br/&gt;
 uid 10050058 timediff 0.943896&lt;br/&gt;
 uid 10031833 timediff 0.942367&lt;br/&gt;
 uid 10061742 timediff 0.941366&lt;br/&gt;
 uid 10062108 timediff 0.940045&lt;br/&gt;
 uid 10064453 timediff 0.939963&lt;br/&gt;
 uid 10069467 timediff 0.937904&lt;br/&gt;
 uid 10030796 timediff 0.937348&lt;br/&gt;
 uid 10075503 timediff 0.933544&lt;br/&gt;
 uid 10077199 timediff 0.932703&lt;br/&gt;
 uid 10058929 timediff 0.932643&lt;br/&gt;
 uid 10054640 timediff 0.932521&lt;br/&gt;
 uid 10034238 timediff 0.931722&lt;br/&gt;
 uid 10062967 timediff 0.929753&lt;br/&gt;
 uid 10041824 timediff 0.928643&lt;br/&gt;
 uid 10037802 timediff 0.928581&lt;br/&gt;
 uid 10067611 timediff 0.927270&lt;br/&gt;
 uid 10073309 timediff 0.927019&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;the slowms is 100ms, but we can&apos;t get any slow log in mongod.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&amp;gt; db.getProfilingStatus()&lt;/p&gt;

&lt;p&gt;&lt;br class=&quot;atl-forced-newline&quot; /&gt;{ &quot;was&quot; : 0, &quot;slowms&quot; : 100 }&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;the&#160;diganostic.data from 3.4.19 were attached too.&lt;/p&gt;</comment>
                            <comment id="2180457" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 02:22:06 +0000"  >&lt;p&gt;yes, awk dicard the decimal point, the real total value may be as:&#160;135259.646036&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;uid 10035044 timediff 1.547164&lt;br/&gt;
uid 10003822 timediff 2966.525757&lt;br/&gt;
uid 10023620 timediff 4055.941557&lt;br/&gt;
uid 10035045 timediff 1.479359&lt;br/&gt;
uid 10053296 timediff 4798.573361&lt;br/&gt;
uid 10073303 timediff 4346.899593&lt;br/&gt;
uid 10003823 timediff 1.621379&lt;br/&gt;
uid 10035046 timediff 1.416700&lt;br/&gt;
uid 10023621 timediff 2.065020&lt;br/&gt;
uid 10013522 timediff 3663.769046&lt;br/&gt;
uid 10063516 timediff 3771.117739&lt;br/&gt;
uid 10053297 timediff 1.573778&lt;br/&gt;
uid 10043645 timediff 3873.324383&lt;br/&gt;
uid 10035047 timediff 1.342320&lt;br/&gt;
uid 10073304 timediff 1.823686&lt;/p&gt;

&lt;p&gt;```&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;i have attached&#160;diagnostic.data from 4.0.6&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2180077" author="eric.sedor" created="Wed, 13 Mar 2019 19:14:18 +0000"  >&lt;p&gt;It is very strange to hear about 4,000ms of latency when no slow operations are logged. Can you also clarify the numbers in the awk output above? Is there a decimal point missing?&#160; If not and this is milliseconds then a lack of slow logged ops is very unexpected.&lt;/p&gt;

&lt;p&gt;For the 4.0.6 deployment, would you please run this test, archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (described &lt;a href=&quot;#full-time-diagnostic-data-capture]&quot; target=&quot;_blank&quot; rel=&quot;noopener&quot;&gt;here&lt;/a&gt;) and attach it to this ticket?&lt;/p&gt;</comment>
                            <comment id="2179438" author="eric.sedor" created="Wed, 13 Mar 2019 14:11:18 +0000"  >&lt;p&gt;Hello and thank you for the detail you&apos;ve provided so far. We are investigating and will let you know if we have additional questions.&lt;/p&gt;</comment>
                            <comment id="2177935" author="stutiredboy@gmail.com" created="Tue, 12 Mar 2019 02:13:44 +0000"  >&lt;p&gt;What we tried:&lt;/p&gt;

&lt;p&gt;10. on MongoDB 3.6.5, compare to &lt;b&gt;normal&lt;/b&gt; results on 3.4.19(about 10xxxx), it has a little higher latency, but stable:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;109008&lt;br/&gt;
113226&lt;br/&gt;
111779&lt;br/&gt;
113159&lt;br/&gt;
111060&lt;br/&gt;
110741&lt;br/&gt;
111174&lt;br/&gt;
109748&lt;br/&gt;
111290&lt;br/&gt;
110055&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;11. on MongoDB 4.0.6, compare to normal results on 3.4.19((about 10xxxx)), it has higher latency, but stable:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;131276&lt;/p&gt;

&lt;p&gt;130429&lt;/p&gt;

&lt;p&gt;127002&lt;/p&gt;

&lt;p&gt;129441&lt;/p&gt;

&lt;p&gt;130086&lt;/p&gt;

&lt;p&gt;129963&lt;/p&gt;

&lt;p&gt;129655&lt;/p&gt;

&lt;p&gt;131308&lt;/p&gt;

&lt;p&gt;131948&lt;/p&gt;

&lt;p&gt;131162&lt;/p&gt;

&lt;p&gt;131298&lt;/p&gt;

&lt;p&gt;131438&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;as our benchmark, we know 3.6 and 4.0 have lower performance than before, so we think the result from 3.6 and 4.0 and normal, without high spike.&lt;/p&gt;</comment>
                            <comment id="2176739" author="stutiredboy@gmail.com" created="Mon, 11 Mar 2019 08:37:48 +0000"  >&lt;blockquote&gt;&lt;p&gt;149726&lt;br/&gt;
 129175&lt;br/&gt;
 124993&lt;br/&gt;
 219767&lt;br/&gt;
 137422&lt;br/&gt;
 156674&lt;br/&gt;
 162410&lt;br/&gt;
 119684&lt;br/&gt;
 117086&lt;br/&gt;
 116885&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;the output above is from:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;while true ; do ./testmongo | awk &apos;{s+=$NF}END{print s}&apos; ; sleep 2 ; done&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;What we tried:&lt;/p&gt;

&lt;p&gt;9. write the test with mongo-cxx-driver r3.3.1 and r3.4.0,&#160; on MongoDB 3.4, they have the same behave: unstable latency with high spike&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="740837">SERVER-40707</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="211416" name="3.4_primary.config.diganostic.data.tgz" size="2608826" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 09:37:48 +0000"/>
                            <attachment id="211417" name="3.4_primary.shard1.diganostic.data.tgz" size="3150461" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 09:37:48 +0000"/>
                            <attachment id="211418" name="3.4_primary.shard2.diganostic.data.tgz" size="2480402" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 09:37:48 +0000"/>
                            <attachment id="211406" name="4.0_primary.config.diagnostic.data.tgz" size="312224" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 02:19:09 +0000"/>
                            <attachment id="211408" name="4.0_primary.shard1.diagnostic.data.tgz" size="387540" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 02:19:09 +0000"/>
                            <attachment id="211407" name="4.0_primary.shard2.diagnostic.data.tgz" size="299158" author="stutiredboy@gmail.com" created="Thu, 14 Mar 2019 02:19:09 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 13 Mar 2019 14:11:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 42 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 42 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>stutiredboy@gmail.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huqeyv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hug2un:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huq187:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>