<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:07:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-44948] High cache pressure in 4.2 replica set vs 4.0</title>
                <link>https://jira.mongodb.org/browse/SERVER-44948</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I have a 3-node replica set running version 4.2.1 on Ubuntu 18.04. Previously we had been on 4.0.4. I&apos;m noticing that the cluster is significantly slower than it was in 4.0.4 both in terms of bulk insert speed on the primary, impact on the speed of other operations while those inserts jobs are running, and the replication lag we see on the secondaries.&lt;/p&gt;

&lt;p&gt;I think at least on the primary this has something to do with elevated cache pressure, but I&apos;m kind of just guessing. I&apos;m happy to provide diagnostic files and logs privately.&lt;/p&gt;

&lt;p&gt;We&apos;re now running with featureCompatibilityVersion 4.2, flow control off, and enableMajorityReadConcern false, and have had some improvement, but we&apos;re still in trouble with cache pressure and insert speed, and the replication lag is quite high when we&apos;re running jobs that insert millions of records.&lt;/p&gt;

&lt;p&gt;Some performance numbers, each with upwards of 15-25m rows:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;With all nodes on 4.0.4: a job averaged 875 rows/sec. The local secondary mostly kept up (maybe 10-60 seconds lag). The geographically remote secondary was never more than 300 seconds behind.&lt;/li&gt;
	&lt;li&gt;With the secondaries on 4.2.1 and the primary on 4.0.4: a job averaged 484 rows/sec. Both secondaries were about 5000-10000 seconds behind by the end of the job.&lt;/li&gt;
	&lt;li&gt;On a similar but standalone node on 4.2.1, the job averaged 4012 rows/sec&lt;/li&gt;
	&lt;li&gt;With all nodes on 4.2.1 and default settings for flowControl and enableMajorityReadConcern: average 265 rows/sec and all other operations were significantly impacted.&lt;/li&gt;
	&lt;li&gt;After disabling flowControl: average 712 rows/sec, but both replicas were about 5000 seconds lagged and all other operations were significantly delayed, for example I saw a document update take 62673ms with&#160;{{timeWaitingMicros: { cache: 62673547 }}}&lt;/li&gt;
	&lt;li&gt;After disabling flowControl and setting enableMajorityReadConcern to false: no specific insert number but I&apos;m seeing &quot;tracked dirty bytes in the cache&quot; at about 1.5 GB while an insert job is running. One secondary is lagged by 4 sec and the other by 1183 sec.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;A couple of notes on our setup:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Our bulk inserts are of ~2.5 KB documents with a large number of indexes (20, one of which is multi-key - array of strings);&lt;/li&gt;
	&lt;li&gt;We do have a couple of change stream listeners (not on the same collection being written to by bulk writes) so some level of majority read concern is still being supported (I suppose)&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="1036929">SERVER-44948</key>
            <summary>High cache pressure in 4.2 replica set vs 4.0</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="mzs">Michael Smith</reporter>
                        <labels>
                    </labels>
                <created>Wed, 4 Dec 2019 20:56:28 +0000</created>
                <updated>Sun, 29 Oct 2023 22:14:26 +0000</updated>
                            <resolved>Tue, 28 Jan 2020 13:58:16 +0000</resolved>
                                    <version>4.2.1</version>
                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="2767759" author="dmitry.agranat" created="Tue, 28 Jan 2020 13:57:40 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzs&quot; class=&quot;user-hover&quot; rel=&quot;mzs&quot;&gt;mzs&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;MongoDB &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/4.2/#jan-27-2020&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;4.2.3&lt;/a&gt; was just released with the fix for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44991&quot; title=&quot;Performance regression in indexes with keys with common prefixes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44991&quot;&gt;&lt;del&gt;SERVER-44991&lt;/del&gt;&lt;/a&gt;. I will go ahead and close this ticket but do not hesitate to reopen if this issue reoccurs. &lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2641379" author="dmitry.agranat" created="Wed, 18 Dec 2019 08:55:09 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzs&quot; class=&quot;user-hover&quot; rel=&quot;mzs&quot;&gt;mzs&lt;/a&gt;, I am checking this information and will update you on the progress.&lt;/p&gt;</comment>
                            <comment id="2636794" author="mzs" created="Tue, 17 Dec 2019 15:58:15 +0000"  >&lt;p&gt;Hi Dmitry,&lt;/p&gt;

&lt;p&gt;It looks like there&apos;s been some progress in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44991&quot; title=&quot;Performance regression in indexes with keys with common prefixes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44991&quot;&gt;&lt;del&gt;SERVER-44991&lt;/del&gt;&lt;/a&gt; and a possible fix has been committed.&lt;/p&gt;

&lt;p&gt;Would you be able to offer any guidance on when it would make it into a 4.2.x patch release? (for example: 2 weeks, 1 month, 2 months...) That will help me come up with a mitigation strategy until then.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Mike&lt;/p&gt;</comment>
                            <comment id="2593384" author="mzs" created="Mon, 9 Dec 2019 12:10:32 +0000"  >&lt;p&gt;Thanks! I&apos;ll keep an eye on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44991&quot; title=&quot;Performance regression in indexes with keys with common prefixes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44991&quot;&gt;&lt;del&gt;SERVER-44991&lt;/del&gt;&lt;/a&gt;. Much appreciated!&lt;/p&gt;</comment>
                            <comment id="2593152" author="dmitry.agranat" created="Mon, 9 Dec 2019 08:47:43 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzs&quot; class=&quot;user-hover&quot; rel=&quot;mzs&quot;&gt;mzs&lt;/a&gt;, with the latest information you&apos;ve provided, we are suspecting this is related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44991&quot; title=&quot;Performance regression in indexes with keys with common prefixes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44991&quot;&gt;&lt;del&gt;SERVER-44991&lt;/del&gt;&lt;/a&gt;. We are still investigating &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44991&quot; title=&quot;Performance regression in indexes with keys with common prefixes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44991&quot;&gt;&lt;del&gt;SERVER-44991&lt;/del&gt;&lt;/a&gt; and I will keep you updated on the progress. &lt;/p&gt;</comment>
                            <comment id="2592371" author="mzs" created="Mon, 9 Dec 2019 00:27:31 +0000"  >&lt;p&gt;I&apos;ve uploaded&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44948&quot; title=&quot;High cache pressure in 4.2 replica set vs 4.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44948&quot;&gt;&lt;del&gt;SERVER-44948&lt;/del&gt;&lt;/a&gt;-perf-prod.zip with more interesting data.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;~22:52:45 - bulk insert started&lt;/li&gt;
	&lt;li&gt;by 00:02:25 one secondary was 600+ seconds lagged, the other about 1000, and small writes to other collections were taking 1-3 seconds with things like timeWaitingMicros: { cache: 1961686 }. Not as bad as I&apos;ve seen it, but I figured it&apos;d be representative.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;(this was with enableFlowControl: false and enableMajorityReadConcern: false)&lt;/p&gt;</comment>
                            <comment id="2592235" author="mzs" created="Sun, 8 Dec 2019 18:26:39 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I&apos;ve uploaded &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44948&quot; title=&quot;High cache pressure in 4.2 replica set vs 4.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44948&quot;&gt;&lt;del&gt;SERVER-44948&lt;/del&gt;&lt;/a&gt;-perf-dev.zip with the perf script output, diagnostic data, and syslog from an insert job in our development environment covering 2019-12-08 17:42:17-17:59:19 with 4.2.1. It may not be interesting - the test file was probably too simple to really slow things down. There was some cache pressure briefly at 2019-12-08 17:50 UTC. The secondaries never got more than about 5 seconds out of sync, even with flow control disabled.&lt;/p&gt;

&lt;p&gt;I&apos;ll run a larger job in prod overnight, but figured I&apos;d send these for now in case they&apos;re sufficient.&lt;/p&gt;

&lt;p&gt;I don&apos;t think I&apos;m able to downgrade a cluster to 4.0.x as we&apos;ve created a number of new indexes since the upgrade. Hopefully something in the dev or prod perf traces from 4.2.1 will be large enough to spot.&lt;/p&gt;

&lt;p&gt;Thanks for taking a look at this.&lt;/p&gt;

&lt;p&gt;Mike&lt;/p&gt;</comment>
                            <comment id="2592140" author="dmitry.agranat" created="Sun, 8 Dec 2019 14:16:58 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzs&quot; class=&quot;user-hover&quot; rel=&quot;mzs&quot;&gt;mzs&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading all the requesting information, it was very useful. We currently suspect a certain regression and will need to collect some additional information.&lt;/p&gt;

&lt;p&gt;In order for us to understand the difference in the reported CPU utilization, we&apos;d like to collect some &lt;tt&gt;perf call stacks&lt;/tt&gt;. This will require the installation of &lt;a href=&quot;https://perf.wiki.kernel.org/index.php/Main_Page&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Linux perf tool&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Before/after comparison would be best, but the suspected regression may be large enough that just data for 4.2.1 would be useful. &lt;/p&gt;

&lt;p&gt;Execute this during the insert workload. This command will capture call stack samples in separate files of 60 seconds each&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;while true; do perf record -a -g -F 99 -o perf.data.$(date -u +%FT%TZ) sleep 60; done&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Then run perf script as above on the subset of files of interest&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for fn in ...; do perf script -i $fn &amp;gt;$fn.txt; done&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Once completed, please upload all of the generated .txt files and a fresh archive of the &lt;tt&gt;diagnostic.data&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Note that it is important to run &lt;tt&gt;perf script&lt;/tt&gt; on the same node where &lt;tt&gt;perf.data&lt;/tt&gt; was generated so that it can be correctly symbolized using the addresses on that machine.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="2587869" author="mzs" created="Fri, 6 Dec 2019 01:56:49 +0000"  >&lt;p&gt;I think we&apos;re in luck, I was able to find diagnostic data and logs from a few trials. I&apos;ve uploaded &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-44948&quot; title=&quot;High cache pressure in 4.2 replica set vs 4.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-44948&quot;&gt;&lt;del&gt;SERVER-44948&lt;/del&gt;&lt;/a&gt;.zip for you.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;2019-11-28 17:32:10 - 19:45:51 UTC: all members 4.0.4, ~1m doc insert job, no significant impact on other operations, no significant lag. (I&apos;d downgraded the secondaries from 4.2.1 back to 4.0.4 before this) See &lt;tt&gt;syslog.7&lt;/tt&gt; for mongod logs.&lt;/li&gt;
	&lt;li&gt;2019-12-01 20:16:47 - 2019-12-02 13:31:33 UTC: all members 4.2.1, fcv 4.2, flowControl on (default), majority read concern enabled (default), ~16m doc insert job, other operations significantly delayed. For example, at 2019-12-02 10:07:34 in syslog.3 I can see a single-document update in a small collection (unrelated to the bulk insert job) took 31032 ms.&lt;/li&gt;
	&lt;li&gt;2019-12-05 22:35:00 UTC until now: all members 4.2.1, fcv 4.2, flowControl off, majority read concern disabled, ~5m doc insert job, significant replication lag, other operations significantly delayed. See &lt;tt&gt;syslog&lt;/tt&gt; for mongod logs.&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="2585379" author="dmitry.agranat" created="Thu, 5 Dec 2019 08:36:50 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mzs&quot; class=&quot;user-hover&quot; rel=&quot;mzs&quot;&gt;mzs&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Thanks for the report. We&apos;d like to compare 4.0.4 and 4.2.1 workloads under default configuration (your example #4). Is it possible to provide these two separate sets of the &lt;tt&gt;diagnostic.data&lt;/tt&gt;? I am asking because the &lt;tt&gt;diagnostic.data&lt;/tt&gt; retention under heavy load (which you&apos;ve described) might only held about a week of data.&lt;/p&gt;

&lt;p&gt;If you still have both 4.0.4 and 4.2.1 data for such comparison, please archive (tar or zip) the &lt;tt&gt;mongod.log&lt;/tt&gt; files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) from all members of the replica set and upload them to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/0e1c3067-f045-4ed9-8463-6d9e31c81a6b.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location. &lt;/p&gt;

&lt;p&gt;Please also note the exact time and timezone of each workload you&apos;d like us to compare.&lt;/p&gt;

&lt;p&gt;If the data from 4.0.4 is no longer available, we might need to consider another approach.&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1040142">SERVER-44991</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1029424">SERVER-44881</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="239727" name="SERVER-44991.svg" size="538684" author="dmitry.agranat@mongodb.com" created="Sun, 8 Dec 2019 19:39:47 +0000"/>
                            <attachment id="239728" name="wt_row_leaf_key_work.png" size="82455" author="dmitry.agranat@mongodb.com" created="Sun, 8 Dec 2019 19:49:12 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 5 Dec 2019 08:36:50 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 2 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 2 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>mzs</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw86w7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvwerr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hw7t5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>