<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:42:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36221] [3.6] Performance regression on small updates to large documents</title>
                <link>https://jira.mongodb.org/browse/SERVER-36221</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The update portion of the repro script&#160;repro-10MBx2.sh from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33191&quot; title=&quot;Cache-full hangs on 3.6&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33191&quot;&gt;&lt;del&gt;SERVER-33191&lt;/del&gt;&lt;/a&gt; shows ~10x regression between 3.6.5 (left) and 3.6.6 (right).&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/192289/192289_regression.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;This seems to be related to eviction of large pages. Interestingly even though we are doing only about 9 updates per second, we are doing 170 evictions per second.&lt;/p&gt;</description>
                <environment></environment>
        <key id="574166">SERVER-36221</key>
            <summary>[3.6] Performance regression on small updates to large documents</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>dmd-perf</label>
                    </labels>
                <created>Fri, 20 Jul 2018 19:00:15 +0000</created>
                <updated>Tue, 6 Dec 2022 03:23:32 +0000</updated>
                            <resolved>Thu, 28 Feb 2019 17:55:29 +0000</resolved>
                                    <version>3.6.6</version>
                                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>21</watches>
                                                                                                                <comments>
                            <comment id="2166488" author="thomas.schubert" created="Thu, 28 Feb 2019 17:55:29 +0000"  >&lt;p&gt;This regression has no longer reproduces after the changes from&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/WT-4483&quot; title=&quot;Improve caching of small updates to large values&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4483&quot;&gt;&lt;del&gt;WT-4483&lt;/del&gt;&lt;/a&gt; were backported into the 3.6 branch. This issue will be resolved in MongoDB 3.6.11.&lt;/p&gt;</comment>
                            <comment id="2079663" author="bruce.lucas@10gen.com" created="Tue, 4 Dec 2018 12:40:19 +0000"  >&lt;p&gt;Thanks Michael. That seems consistent with the following metrics for four runs (3.6.5, 3.6.6, 3.6.8, 4.0.2).&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/202582/202582_evictions.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I think that the &quot;pages evicted because they exceeded the in-memory maximum count&quot; refers to pages evicted due to exceeding the maximum number of deltas as you suggested.&lt;/p&gt;

&lt;p&gt;We also see as you suggested the oldest_timestamp being updated less often. Interestingly the rate at which the oldest timestamp is updated is about the same as the rate of log flushes, so perhaps there is something that connects them? The change that caused this regression (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt;) reduced the rate of log flushes so if that also reduced the rate at which oldest_timestamp is updated that could explain the regression.&lt;/p&gt;

&lt;p&gt;In the 4.0.4 run however we see the rate of log flushes is as high as it was in 3.6.5 before &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt;. So if the above analysis is correct that means that the reason we don&apos;t see the regression in 4.0 is that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt; is for some reason ineffective in 4.0. Maybe the reason is that there are a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b0d400815c3159b4cb30b6818138fc55036f8ed4/src/mongo/db/storage/wiredtiger/wiredtiger_oplog_manager.cpp#L182&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;couple of conditions&lt;/a&gt; under which we don&apos;t reduce the rate of flushes, and maybe that is more often the case in 4.0 than 3.6?&lt;/p&gt;</comment>
                            <comment id="2079486" author="michael.cahill" created="Tue, 4 Dec 2018 05:51:15 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=brian.lane&quot; class=&quot;user-hover&quot; rel=&quot;brian.lane&quot;&gt;brian.lane&lt;/a&gt; pinged me about this ticket.&lt;/p&gt;

&lt;p&gt;This workload is sensitive to two factors:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;how large a page is allowed to grow in cache before it is forcibly evicted (by default this is 10MB so the reproducer creates a page that is permanently over this limit); and&lt;/li&gt;
	&lt;li&gt;how many delta updates accumulate before making a full copy of the document (default 10, as long as one of the updates is stable, i.e., older than &lt;tt&gt;oldest_timestamp&lt;/tt&gt;).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The WT code for both of these is effectively the same between v3.6 and v4.0, but both are sensitive to how frequently the &lt;tt&gt;oldest_timestamp&lt;/tt&gt; is updated. That&apos;s because WT won&apos;t retry eviction for a page if &lt;tt&gt;oldest_timestamp&lt;/tt&gt; hasn&apos;t change since last time it tried, and because we keep buffering delta changes if the &lt;tt&gt;oldest_timestamp&lt;/tt&gt; hasn&apos;t moved forward. Given this, I suspect some difference in replication is involved, and backporting that change may not be desirable in order to address this performance regression.&lt;/p&gt;

&lt;p&gt;Instead, we could consider allowing pages containing large documents to grow larger and/or allowing more delta updates to accumulate before making full copies of documents. The former trades larger pages in cache for less frequent eviction, the latter will make some reads slower (because they have to apply more delta updates) in order to make the updates faster.&lt;/p&gt;</comment>
                            <comment id="2019023" author="bruce.lucas@10gen.com" created="Mon, 1 Oct 2018 14:04:03 +0000"  >&lt;p&gt;After a little investigation here&apos;s some more information (some of this confirms or repeats comments from above);&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;The script stops one of the secondaries, and the regression is much less if one of the secondaries is not stopped. Following table shows time taken for the update phase of the script that shows the regression:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        one secondary    both secondaries&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        running          running&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3.6.5     28 sec          26 sec&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3.6.6    183 sec          41 sec&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;I suspect this is because the change in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt; only applies if there are no oplog waiters, and with only one secondary running this is a much larger fraction of the time. I believe the same regression would be seen if both secondaries were running but lagged because they would not be waiting for the oplog as often.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;The regression appears to be the result of doing far more evictions. As noted in the opening comment in 3.6.6 we are doing about 170 evictions per second (&quot;pages evicted because they exceeded the in-memory maximum&quot;) even though we are only doing 9 updates per second. Stack traces taken during a run show the same story:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3.6.5:  4 of 13 (31%) of CmdUpdate are in __wt_page_release_evict&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3.6.6: 41 of 43 (95%) of CmdUpdate are in __wt_page_release_evict&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;So the questions seem to be&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;why does the change made in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt; have the side effect of causing far more evictions?&lt;/li&gt;
	&lt;li&gt;why does the same issue not occur in 4.0?&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2018058" author="bruce.lucas@10gen.com" created="Fri, 28 Sep 2018 19:45:00 +0000"  >&lt;p&gt;A git bisect identified the following commit as the cause of this regression:&lt;/p&gt;

&lt;p&gt;a5656bc506fcdfe36d44cfb3c17969cbe35695c0 is the first bad commit&lt;br/&gt;
&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31679&quot; title=&quot;Increase in disk i/o for writes to replica set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31679&quot;&gt;&lt;del&gt;SERVER-31679&lt;/del&gt;&lt;/a&gt; Limit journaling frequency without oplog waiters&lt;/p&gt;</comment>
                            <comment id="2009337" author="bruce.lucas@10gen.com" created="Thu, 20 Sep 2018 15:51:49 +0000"  >&lt;p&gt;It appears that this regression is fixed in 4.0.2 (but not 3.6.8), and in fact performance is both improved and less erratic, on the two workloads attached to this ticket:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;repro-10MBx2-regression.sh&lt;/b&gt;&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/196757/196757_four.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&lt;b&gt;repro-10MBx2-erratic-1.sh&lt;/b&gt;&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/196758/196758_four-erratic.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Each chart shows four runs, marked A-D, using 3.6.5, 3.6.6, 3.6.8, and 4.0.2 respectively.&lt;/p&gt;</comment>
                            <comment id="1961439" author="benety.goh" created="Tue, 31 Jul 2018 15:20:44 +0000"  >&lt;p&gt;moving code review from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36221&quot; title=&quot;[3.6] Performance regression on small updates to large documents&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36221&quot;&gt;&lt;del&gt;SERVER-36221&lt;/del&gt;&lt;/a&gt; to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36373&quot; title=&quot;create test to fill WT cache during steady state replication&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36373&quot;&gt;&lt;del&gt;SERVER-36373&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1956965" author="alexander.gorrod" created="Thu, 26 Jul 2018 01:01:02 +0000"  >&lt;p&gt;I&apos;d start by looking at whether &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4105&quot; title=&quot;Optimize cache usage for update workload with history pinned&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4105&quot;&gt;&lt;del&gt;WT-4105&lt;/del&gt;&lt;/a&gt; caused the change in behavior - it&apos;s directly related.&lt;/p&gt;</comment>
                            <comment id="1956379" author="bruce.lucas@10gen.com" created="Wed, 25 Jul 2018 16:10:27 +0000"  >&lt;p&gt;Attached is a simpler script, repro-10MBx2-regression.sh, that shows the regression in 3.6.6. Also attached stacks-regression.txt which look like they show the update often in __rec_split_write.&lt;/p&gt;

&lt;p&gt;That script stops one of the secondaries while the updates are being applied. Interestingly the regression disappears if the secondary is not stopped. However, depending on factors like document size and number of documents, the throughput under 3.6.6 even with all members of the replica set in operation can be very erratic, e.g.:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/192629/192629_erratic-1.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;3.6.5 on the left, 3.6.6 on the right. Attached repro script repro-10MBx2-erratic-1.sh and ftdc data erratic-1.tar.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="646470">WT-4483</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="646470">WT-4483</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="567709">SERVER-35958</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="449479">SERVER-31679</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="494491">SERVER-33191</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="580291">SERVER-36373</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="552075">WT-4105</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="192629" name="erratic-1.png" size="48806" author="bruce.lucas@mongodb.com" created="Wed, 25 Jul 2018 16:08:41 +0000"/>
                            <attachment id="192631" name="erratic-1.tar" size="188928" author="bruce.lucas@mongodb.com" created="Wed, 25 Jul 2018 16:09:04 +0000"/>
                            <attachment id="202582" name="evictions.png" size="109153" author="bruce.lucas@mongodb.com" created="Tue, 4 Dec 2018 12:39:23 +0000"/>
                            <attachment id="196758" name="four-erratic.png" size="51219" author="bruce.lucas@mongodb.com" created="Thu, 20 Sep 2018 15:49:04 +0000"/>
                            <attachment id="196757" name="four.png" size="47757" author="bruce.lucas@mongodb.com" created="Thu, 20 Sep 2018 15:48:55 +0000"/>
                            <attachment id="192289" name="regression.png" size="108585" author="bruce.lucas@mongodb.com" created="Fri, 20 Jul 2018 19:00:13 +0000"/>
                            <attachment id="192633" name="repro-10MBx2-erratic-1.sh" size="1209" author="bruce.lucas@mongodb.com" created="Wed, 25 Jul 2018 16:09:16 +0000"/>
                            <attachment id="192632" name="repro-10MBx2-regression.sh" size="1202" author="bruce.lucas@mongodb.com" created="Wed, 25 Jul 2018 16:09:16 +0000"/>
                            <attachment id="192630" name="stacks-regression.txt" size="1971369" author="bruce.lucas@mongodb.com" created="Wed, 25 Jul 2018 16:09:05 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 23 Jul 2018 15:30:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 49 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WT-4483'>WT-4483</a></s>, <s><a href='https://jira.mongodb.org/browse/BACKPORT-3863'>BACKPORT-3863</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 49 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>benety.goh@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu336f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr7i5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2394">Storage NYC 2018-07-30</customfieldvalue>
    <customfieldvalue id="2455">Storage NYC 2018-08-13</customfieldvalue>
    <customfieldvalue id="2456">Storage NYC 2018-08-27</customfieldvalue>
    <customfieldvalue id="2490">Storage NYC 2018-09-10</customfieldvalue>
    <customfieldvalue id="2586">Storage NYC 2018-10-22</customfieldvalue>
    <customfieldvalue id="2595">Storage NYC 2018-11-05</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu2pfr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>