<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:20:28 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29311] WT performance drops to zero during cache eviction</title>
                <link>https://jira.mongodb.org/browse/SERVER-29311</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi all,&lt;/p&gt;

&lt;p&gt;We are currently benchmarking our workload against the following test environment:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;MongoDB 3.2.12 (also tested on 3.2.11)&lt;/li&gt;
	&lt;li&gt;Sharded cluster (2 shards, pre-chunked, 1 shard has 1 node and the other has 2 nodes in a replica set)&lt;/li&gt;
	&lt;li&gt;Each node has 2x20 core CPUs and 32GB of memory&lt;/li&gt;
	&lt;li&gt;WiredTiger storage engine&lt;/li&gt;
	&lt;li&gt;We are using 2 mongos&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Our synthetic workload consists of 450 workers performing an insert, waiting for 1 to 100ms and then updating the same document (adding roughly 1k bytes) querying by the &lt;tt&gt;_id&lt;/tt&gt; field. We routinely see throughput drops to 0 for both inserts and updates.&lt;/p&gt;

&lt;p&gt;We&apos;ve been collecting various stats while running the workload generator, and have correlated the drops to 0 throughput with a declining TrackedDirtyBytesInTheCache delta (measured every second using db.serverStatus()) which is what led us to believe it was related to the cache eviction. &lt;/p&gt;

&lt;p&gt;During periods of 0 throughput, the mongod boxes become CPU bound (40 core boxes) - this is what led us to conclude mongo was stalling as opposed to anything on application side. See the attached graph which is a plot of throughput vs. mongod CPU usage.&lt;/p&gt;

&lt;p&gt;After resizing the wiredtiger cache from 19GB to 1GB (with wiredTiger.engineConfig.cacheSizeGB=1) the throughput drops are much shorter in duration but still noticeable. We also tested setting eviction_dirty_trigger=10 and eviction_dirty_target=7 (with a 19GB cache) as suggested in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28488&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-28488&lt;/a&gt; but we still see periods of 0 throughput.&lt;/p&gt;

&lt;p&gt;I have uploaded diagnostic data from a mongod instance previously under ticket #28488, and below are a couple of links to various stats we&apos;ve recorded during various (different) testing runs in the hope they help. I can run tests and provide any info needed over the next couple of weeks. Sorry for the dupe ticket, but it has been over a month without a response.&lt;/p&gt;

&lt;p&gt;To clarify, our workload generator is the only application running on a 40 CPU core box, connected via a 1Gbps LAN to mongos - it performs no IO other than talking to mongo, it&apos;s sole purpose is to load test mongo. Monitoring the network interface, we never come remotely close to maxing the link, and we&apos;re nowhere near maxing the CPU of the host at any point - we are not I/O bound as claimed in the other ticket.&lt;/p&gt;

&lt;p&gt;CPU + disk IO stats: &lt;a href=&quot;https://docs.google.com/a/itsallbroken.com/spreadsheets/d/1o6N23E_Nc1dBnFC_XXs6qWAXcdKAGjlgRuGHuVEXzn0/edit?usp=sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/a/itsallbroken.com/spreadsheets/d/1o6N23E_Nc1dBnFC_XXs6qWAXcdKAGjlgRuGHuVEXzn0/edit?usp=sharing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;db.serverStatus() + TrackedDirtyBytesInTheCache plot (switch sheets): &lt;a href=&quot;https://docs.google.com/a/itsallbroken.com/spreadsheets/d/1tsK8vOun4IHyfhpA7mXQ9GrO9zneoWvL2dENj2EImug/edit?usp=sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/a/itsallbroken.com/spreadsheets/d/1tsK8vOun4IHyfhpA7mXQ9GrO9zneoWvL2dENj2EImug/edit?usp=sharing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/154467/Screen%20Shot%202017-04-21%20at%2014.23.35.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/154468/Screen%20Shot%202017-04-21%20at%2014.47.22.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment>FreeBSD 11, sharded, replicated.</environment>
        <key id="385494">SERVER-29311</key>
            <summary>WT performance drops to zero during cache eviction</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="domodwyer">Dom Dwyer</reporter>
                        <labels>
                    </labels>
                <created>Sun, 21 May 2017 18:57:23 +0000</created>
                <updated>Thu, 30 Nov 2017 14:32:03 +0000</updated>
                            <resolved>Thu, 29 Jun 2017 17:30:51 +0000</resolved>
                                    <version>3.2.11</version>
                    <version>3.2.12</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="1610641" author="thomas.schubert" created="Thu, 29 Jun 2017 17:30:51 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=domodwyer&quot; class=&quot;user-hover&quot; rel=&quot;domodwyer&quot;&gt;domodwyer&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=weishan&quot; class=&quot;user-hover&quot; rel=&quot;weishan&quot;&gt;weishan&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve examined your diagnostic.data and do not see a clear indication of a bug in WiredTiger.&lt;/p&gt;

&lt;p&gt;Please note that we are continuing to work to improve WiredTiger for an assortment of workloads and it&apos;s likely that you will benefit from some of these changes. For example, &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3345&quot; title=&quot;Improve rwlock scaling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3345&quot;&gt;&lt;del&gt;WT-3345&lt;/del&gt;&lt;/a&gt;, may be exacerbating the issues that you&apos;re observing. Therefore, I would recommend staying current with the latest version of MongoDB in your test environment.&lt;/p&gt;

&lt;p&gt;However, as described previously in this ticket, the root issue appears to be a configuration problem around an I/O bottleneck. Therefore, I would recommend posting  on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; for MongoDB-related support. See also our &lt;a href=&quot;https://docs.mongodb.org/manual/support&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Technical Support page&lt;/a&gt; for additional support resources.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1604118" author="weishan" created="Thu, 22 Jun 2017 09:48:33 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;We found this issue(&lt;a href=&quot;https://jira.mongodb.org/browse/WT-2705&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/WT-2705&lt;/a&gt;) that seems similar to what we are experiencing as well. We observe the same behaviour of high latency and CPU utilisation during cache eviction in our test as well.&lt;/p&gt;

&lt;p&gt;Do you think it is the same issue?&lt;/p&gt;</comment>
                            <comment id="1601891" author="domodwyer" created="Tue, 20 Jun 2017 17:24:29 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;Thanks for the pointer regarding the slow ZFS writes, we had overlooked tuning our testing environment for the MongoDB workload (specifically the filesystem block size).&lt;/p&gt;

&lt;p&gt;Unfortunately even after tuning ZFS we&apos;re still seeing 0 throughput around ~18% of the time using the default eviction settings. When using no WT compression and the settings you suggest, we see 0 ops ~6% of the time but overall throughput is nearly halved and the 99th percentile update latency jumps to close to half a second - see #1. Same results when using snappy for compression too. The large write latency distribution and low throughput prevents us running with these settings in production.&lt;/p&gt;

&lt;p&gt;We&apos;ve run benchmarks against all 6 combinations of compression settings (WT with zlib, snappy, and none; ZFS with lz4 on and off) across various filesystem block sizes and we&apos;ve seen the issue occur in all of them. These tests were run with the default eviction trigger and eviction targets to try and isolate just the compression setting impact. Our raw data for these tests can be found at #2, though note some tests have double the sample count than others.&lt;/p&gt;

&lt;p&gt;It seems compression isn&apos;t a factor here, is there anything else we can try? Have you managed to reproduce this?&lt;/p&gt;

&lt;p&gt;I have the diagnostic.data from a run with your suggested eviction settings but it&apos;s too big to attach to this ticket (~200MB), is there somewhere else I should upload it to? If you need any of the raw benchmark logs or any other data please let me know.&lt;/p&gt;

&lt;p&gt;EDIT: &lt;a href=&quot;https://itsallbroken.com/diagnostic.data_16K.tar&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://itsallbroken.com/diagnostic.data_16K.tar&lt;/a&gt;&lt;/p&gt;

&lt;hr /&gt;
&lt;p&gt;#1: &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1JqOF3AvXKciEtHiTMtD9tgWzTzNqUIRyC0vMgpx5LKs/edit#gid=43245115&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/spreadsheets/d/1JqOF3AvXKciEtHiTMtD9tgWzTzNqUIRyC0vMgpx5LKs/edit#gid=43245115&lt;/a&gt;&lt;br/&gt;
#2: &lt;a href=&quot;https://docs.google.com/spreadsheets/d/1S7x7AtJuY5AqxLVYVLBdBK33UiuMlXnEtyS0972UvxA/edit#gid=166349404&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/spreadsheets/d/1S7x7AtJuY5AqxLVYVLBdBK33UiuMlXnEtyS0972UvxA/edit#gid=166349404&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1593316" author="thomas.schubert" created="Fri, 9 Jun 2017 18:58:25 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=weishan&quot; class=&quot;user-hover&quot; rel=&quot;weishan&quot;&gt;weishan&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=domodwyer&quot; class=&quot;user-hover&quot; rel=&quot;domodwyer&quot;&gt;domodwyer&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It appears that writes are slow to ZFS. In the diagnostic.data, we see a larger number of active filesystem write calls this makes the single-threaded part of checkpoints slow, which in turn throttles application writes.&lt;/p&gt;

&lt;p&gt;One thing that contributes to this is using &quot;blockCompressor: zlib&quot; &amp;#8211; my first suggestion is to switch to snappy. I&apos;d also suggest testing your workload with the following settings:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongod ... --wiredTigerEngineConfigString=&quot;eviction_dirty_trigger=2,eviction_dirty_target=5,eviction_checkpoint_target=2&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will reduce the amount of dirty data in cache meaning checkpoints complete faster, which should reduce the impact of slow I/O on your application.&lt;/p&gt;

&lt;p&gt;Would you please implement these changes and let us know if it resolves the issue?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1590866" author="weishan" created="Thu, 8 Jun 2017 13:01:51 +0000"  >&lt;p&gt;Hello!&lt;/p&gt;

&lt;p&gt;Are there anything else you need from us in order to reproduce the issue at your end?&lt;/p&gt;

&lt;p&gt;WeiShan&lt;/p&gt;</comment>
                            <comment id="1578769" author="domodwyer" created="Wed, 24 May 2017 13:02:39 +0000"  >&lt;p&gt;Hi all,&lt;/p&gt;

&lt;p&gt;Attached is a graph from the latest run against 3.2.13 (the diagnostic.data collection run). Inserts/s are measured from workload generator running on it&apos;s own box, the other stats are measured from the master of the replicated nodes.&lt;/p&gt;

&lt;p&gt;With 3.2.13 we see less periods of being CPU bound, though we still see throughput issues. They seem to fit with the cache size deltas changing still.&lt;/p&gt;

&lt;p&gt;I don&apos;t have the original data (DB engineer sent over the attached graph) but I&apos;m sure I can get hold of a copy if you think it&apos;ll be of use. We can run tests if needed, but we will be using our test environment for other things in the coming weeks so it might take a few days.&lt;/p&gt;

&lt;p&gt;Dom&lt;/p&gt;</comment>
                            <comment id="1577862" author="domodwyer" created="Tue, 23 May 2017 16:20:21 +0000"  >&lt;p&gt;Attached is the diagnostic data as promised.&lt;/p&gt;

&lt;p&gt;We can confirm we see the same issue with 3.2.13, however the periods of 0 throughput are less drastic. I&apos;ve also attached a throughput log from the latest run.&lt;/p&gt;</comment>
                            <comment id="1576988" author="thomas.schubert" created="Mon, 22 May 2017 19:10:44 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=domodwyer&quot; class=&quot;user-hover&quot; rel=&quot;domodwyer&quot;&gt;domodwyer&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Please attach the diagnostic.data to this ticket.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1576799" author="domodwyer" created="Mon, 22 May 2017 16:36:41 +0000"  >&lt;p&gt;Hi Michael,&lt;/p&gt;

&lt;p&gt;Thanks for the quick reply. We&apos;re going to push 3.2.13 to our staging environment and re-run the tests. &lt;/p&gt;

&lt;p&gt;We should have the diagnostic data for you tomorrow. Should I attach it here, or upload it to &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/423c6cab-ec2f-47a2-9df3-3bc45f922d2c.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/423c6cab-ec2f-47a2-9df3-3bc45f922d2c.html&lt;/a&gt; ?&lt;/p&gt;

&lt;p&gt;Dom&lt;/p&gt;</comment>
                            <comment id="1576281" author="michael.cahill" created="Sun, 21 May 2017 23:37:19 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=domodwyer&quot; class=&quot;user-hover&quot; rel=&quot;domodwyer&quot;&gt;domodwyer&lt;/a&gt;, thanks for the report, sorry you are seeing this issue.&lt;/p&gt;

&lt;p&gt;Can you please attach a tarball containing the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory from a run of the workload?  Also, if possible, can you please repeat with MongoDB 3.2.13, which avoids some fsync calls that were present in earlier versions and adds additional statistics that may help identify the issue?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="367613">SERVER-28488</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="156802" name="diagnostic.data_mongo3213.tar" size="129777664" author="domodwyer" created="Tue, 23 May 2017 16:06:07 +0000"/>
                            <attachment id="156633" name="mongo1.png" size="556480" author="domodwyer" created="Mon, 22 May 2017 08:46:02 +0000"/>
                            <attachment id="156632" name="mongo2.png" size="796696" author="domodwyer" created="Mon, 22 May 2017 08:46:03 +0000"/>
                            <attachment id="156803" name="output_mongo3213_trim.log" size="74687" author="domodwyer" created="Tue, 23 May 2017 16:26:26 +0000"/>
                            <attachment id="156872" name="run2-cache-activity.png" size="429776" author="domodwyer" created="Wed, 24 May 2017 12:58:31 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 21 May 2017 23:37:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 32 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 32 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>domodwyer</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>weishan</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht7rdb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hszvin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Configure a large cache, perform many insert+update operations, measuring throughput.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht7dfr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>