<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:29:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32139] Oplog truncation creates large amount of dirty data in cache</title>
                <link>https://jira.mongodb.org/browse/SERVER-32139</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;1-node repl set, HDD, 500 GB oplog, 4 GB cache, 5 threads inserting 130 kB docs. Left is 3.6.0-rc8, right is 3.4.10:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/172171/172171_comparison.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;At the point A where the oplog fills and oplog truncation begins in 3.6.0-rc8 we see large amounts of oplog being read into cache and large amounts dirty data in cache and resulting operation stalls. This does not occur at the corresponding point B when running on 3.4.10.&lt;/p&gt;

&lt;p&gt;FTDC data for the above two runs attached.&lt;/p&gt;</description>
                <environment></environment>
        <key id="467003">SERVER-32139</key>
            <summary>Oplog truncation creates large amount of dirty data in cache</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="luke.chen@mongodb.com">Luke Chen</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>RF36</label>
                    </labels>
                <created>Fri, 1 Dec 2017 18:05:55 +0000</created>
                <updated>Thu, 1 Mar 2018 15:27:12 +0000</updated>
                            <resolved>Wed, 28 Feb 2018 03:54:26 +0000</resolved>
                                    <version>3.6.0-rc8</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>23</watches>
                                                                                                                <comments>
                            <comment id="1818140" author="luke.chen" created="Wed, 28 Feb 2018 03:54:26 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3805&quot; title=&quot;Avoid reading lookaside pages in truncate fast path&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3805&quot;&gt;&lt;del&gt;WT-3805&lt;/del&gt;&lt;/a&gt; addressed the issue recorded in this ticket. Close this ticket as a dup. &lt;/p&gt;</comment>
                            <comment id="1818121" author="luke.chen" created="Wed, 28 Feb 2018 03:32:19 +0000"  >&lt;p&gt;I re-ran the repro.sh script on a ubuntu machine (with SSD) with binary of the latest MongoDB master (git version: acd27aa33043cd1bdf94e9e8017001e0ea46a41a) which includes the changes from &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3805&quot; title=&quot;Avoid reading lookaside pages in truncate fast path&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3805&quot;&gt;&lt;del&gt;WT-3805&lt;/del&gt;&lt;/a&gt;. This time &quot;ss wt cache dirty fill ratio&quot; remained the same level after the oplog truncation started. The diagnostic.data archive &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/180433/180433_dd_wt3805.tgz&quot; title=&quot;dd_wt3805.tgz attached to SERVER-32139&quot;&gt;dd_wt3805.tgz&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; is uploaded.&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/180432/180432_master_with_wt-3805.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                            <comment id="1816833" author="michael.cahill" created="Tue, 27 Feb 2018 04:53:03 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=luke.chen&quot; class=&quot;user-hover&quot; rel=&quot;luke.chen&quot;&gt;luke.chen&lt;/a&gt;, now that &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3805&quot; title=&quot;Avoid reading lookaside pages in truncate fast path&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3805&quot;&gt;&lt;del&gt;WT-3805&lt;/del&gt;&lt;/a&gt; has been merged into master, can you please re-run &lt;tt&gt;repro.sh&lt;/tt&gt;, gather the new diagnostic data and confirm that this issue is resolved?  (That is, confirm that we see behavior like MongoDB 3.4.x, not 3.6.0).&lt;/p&gt;</comment>
                            <comment id="1752756" author="neha.khatri" created="Fri, 15 Dec 2017 02:41:56 +0000"  >&lt;p&gt;Attaching the diagnostic data obtained from the repro script execution on MongoDB Master and on master with &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3767&quot; title=&quot;Avoid lookaside instantiation for faster reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3767&quot;&gt;&lt;del&gt;WT-3767&lt;/del&gt;&lt;/a&gt;+&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3768&quot; title=&quot;Lookaside optimization: birthmarks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3768&quot;&gt;&lt;del&gt;WT-3768&lt;/del&gt;&lt;/a&gt; &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/173593/173593_dd_mdb_master.zip&quot; title=&quot;dd_mdb_master.zip attached to SERVER-32139&quot;&gt;dd_mdb_master.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/173594/173594_dd.wt3767_68.zip&quot; title=&quot;dd.wt3767_68.zip attached to SERVER-32139&quot;&gt;dd.wt3767_68.zip&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; . &lt;/p&gt;</comment>
                            <comment id="1752698" author="alexander.gorrod" created="Fri, 15 Dec 2017 00:16:01 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=neha.khatri&quot; class=&quot;user-hover&quot; rel=&quot;neha.khatri&quot;&gt;neha.khatri&lt;/a&gt; I have created &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3805&quot; title=&quot;Avoid reading lookaside pages in truncate fast path&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3805&quot;&gt;&lt;del&gt;WT-3805&lt;/del&gt;&lt;/a&gt; that outlines an optimization that should address the issue of truncate pulling a lot of content into cache.. We should rerun the workload to confirm once that ticket is resolved.&lt;/p&gt;</comment>
                            <comment id="1751586" author="neha.khatri" created="Thu, 14 Dec 2017 02:48:56 +0000"  >&lt;p&gt;I executed the repro script on the MongoDB master and on MongoDB with &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3767&quot; title=&quot;Avoid lookaside instantiation for faster reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3767&quot;&gt;&lt;del&gt;WT-3767&lt;/del&gt;&lt;/a&gt;,&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3768&quot; title=&quot;Lookaside optimization: birthmarks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3768&quot;&gt;&lt;del&gt;WT-3768&lt;/del&gt;&lt;/a&gt; changes. Both the runs show high &lt;tt&gt;cache dirty fill ratio&lt;/tt&gt; after the oplog truncation start. &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/173474/173474_master_vs_wt3767_3768.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;The problem persists even after applying the above two changes.&lt;/p&gt;</comment>
                            <comment id="1750463" author="alexander.gorrod" created="Wed, 13 Dec 2017 03:42:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=neha.khatri&quot; class=&quot;user-hover&quot; rel=&quot;neha.khatri&quot;&gt;neha.khatri&lt;/a&gt; It would be great if you can reproduce this with the latest version of MongoDB master, and then re-run with the changes from &lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/pull/3813&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WT-3767&lt;/a&gt; and &lt;a href=&quot;https://github.com/wiredtiger/wiredtiger/pull/3812&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WT-3768&lt;/a&gt; applied to see if it resolves or helps the issue.&lt;/p&gt;</comment>
                            <comment id="1743182" author="daniel.gottlieb@10gen.com" created="Tue, 5 Dec 2017 17:30:56 +0000"  >&lt;p&gt;Copying some analysis over:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I can confirm this benchmark can cause bubbles where the insert rate falls sharply for ~2-3 seconds correlating with a slow call to WT&apos;s &lt;tt&gt;log_flush&lt;/tt&gt;. I&apos;m not sure if that call does some locking in addition to IO that requires finer grained measurement.&lt;/p&gt;

&lt;p&gt;Typically the journal flush takes 1-2 milliseconds. The bubble occurs when successive calls take 479, 2183, 1821 and 149 milliseconds. Assuming a 1-1 mapping of optime &quot;incs&quot; to inserts, the typical log flush advances the oplog read timestamp (not the oldest timestamp, but related) ~1000 documents. The large flush times advanced the oplog read timestamp ~200,000 documents.&lt;/p&gt;&lt;/blockquote&gt;</comment>
                            <comment id="1743077" author="milkie" created="Tue, 5 Dec 2017 16:38:14 +0000"  >&lt;p&gt;Yes, waiting for the log_flush call to complete.  &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; has just confirmed this with his own local testing.&lt;/p&gt;</comment>
                            <comment id="1743006" author="bruce.lucas@10gen.com" created="Tue, 5 Dec 2017 16:13:53 +0000"  >&lt;p&gt;The following chart shows the lag in the optime metrics from replSetGetStatus - each &quot;x lag&quot; metric is the difference between that optime and the applied optime.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/172424/172424_optimes.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;In particular it shows that indeed our durable optime lag exactly mirrors the range of pinned timestamps. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; does this confirm that the delay in updating the oldest timestamp is due to waiting for durability?&lt;/p&gt;</comment>
                            <comment id="1742800" author="milkie" created="Tue, 5 Dec 2017 14:25:21 +0000"  >&lt;p&gt;We need to keep timestamps around long enough for oplog visibility to work, and oplog visiblity depends on log_flush.  We might be able to do something complicated but it would be only for one-node replica sets.&lt;/p&gt;</comment>
                            <comment id="1742508" author="alexander.gorrod" created="Tue, 5 Dec 2017 04:52:19 +0000"  >&lt;p&gt;There is work scheduled in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3767&quot; title=&quot;Avoid lookaside instantiation for faster reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3767&quot;&gt;&lt;del&gt;WT-3767&lt;/del&gt;&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3768&quot; title=&quot;Lookaside optimization: birthmarks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3768&quot;&gt;&lt;del&gt;WT-3768&lt;/del&gt;&lt;/a&gt; that should reduce the overhead of using the lookaside file seen here.&lt;/p&gt;

&lt;p&gt;Ideally MongoDB would update the oldest timestamp more regularly to avoid using the lookaside file at all in this situation - &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; do you think that will be possible, or is the &lt;tt&gt;log_flush&lt;/tt&gt; the smallest unit of work required to update the timestamp?&lt;/p&gt;</comment>
                            <comment id="1741615" author="bruce.lucas@10gen.com" created="Mon, 4 Dec 2017 16:41:49 +0000"  >&lt;p&gt;This test is bottlenecked on disk, so yes it&apos;s conceivable that an fsync could block for some time, and/or there could be a delay in the WT log implementation. It should be simple enough to add some instrumentation to see if that is the case.&lt;/p&gt;</comment>
                            <comment id="1741526" author="milkie" created="Mon, 4 Dec 2017 15:52:31 +0000"  >&lt;p&gt;Along the same lines as my earlier idea, perhaps the thread that updates MongoDB&apos;s all_committed time is blocked waiting for its log_flush(&quot;sync=on&quot;) call to return.  Would there be a situation where this call would block for long periods?&lt;/p&gt;</comment>
                            <comment id="1741476" author="bruce.lucas@10gen.com" created="Mon, 4 Dec 2017 15:18:14 +0000"  >&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/172269/172269_correlations.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt;, I&apos;m indeed seeing good correlation between range of timestamps pinned and lookaside activitity in my repro.&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, I can&apos;t find evidence of stalled inserts in the server log. For example, at A the timestamp hasn&apos;t been updated for about 5 seconds, but we don&apos;t see any slow inserts logged (&quot;logged slowest query durations&quot;).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I spotted some other possibly interesting correlations with lookaside activity, significance tbd.&lt;/p&gt;</comment>
                            <comment id="1741382" author="milkie" created="Mon, 4 Dec 2017 13:40:57 +0000"  >&lt;p&gt;One possibility for why the oldest_timestamp lags sometimes is that the all_committed time isn&apos;t moving ahead &amp;#8211; possibly due to one lagging transaction.  But that would mean one of the 5 threads would have stalled.&lt;/p&gt;</comment>
                            <comment id="1741380" author="bruce.lucas@10gen.com" created="Mon, 4 Dec 2017 13:34:27 +0000"  >&lt;p&gt;I&apos;ve attached a self-contained repro script. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt;, differences from yours:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;4 GB cache&lt;/li&gt;
	&lt;li&gt;considerably higher rate of inserts, possibly because the workload is i/o bound and my repro uses more compressible documents&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I suspect net result is higher cache pressure which gives a stronger signal for the issue.&lt;/p&gt;</comment>
                            <comment id="1740988" author="bruce.lucas@10gen.com" created="Sun, 3 Dec 2017 14:54:31 +0000"  >&lt;p&gt;The same effect is seen with a 50 GB oplog and SSD (vs 500 GB oplog and HDD for the test above). Starting at A where the oplog rolls over and truncation begins we see a large amount of dirty data and operation stalls.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/172230/172230_smalloplog.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="471993">WT-3805</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="478899">SERVER-32547</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="467495">SERVER-32154</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="172247" name="Screen Shot 2017-12-04 at 8.01.26 pm.png" size="488196" author="alexander.gorrod@mongodb.com" created="Mon, 4 Dec 2017 09:04:12 +0000"/>
                            <attachment id="172248" name="Screen Shot 2017-12-04 at 8.14.44 pm.png" size="281947" author="alexander.gorrod@mongodb.com" created="Mon, 4 Dec 2017 09:15:23 +0000"/>
                            <attachment id="172171" name="comparison.png" size="170564" author="bruce.lucas@mongodb.com" created="Fri, 1 Dec 2017 18:06:47 +0000"/>
                            <attachment id="172269" name="correlations.png" size="368250" author="bruce.lucas@mongodb.com" created="Mon, 4 Dec 2017 15:16:57 +0000"/>
                            <attachment id="172173" name="dd-3410.tar" size="970752" author="bruce.lucas@mongodb.com" created="Fri, 1 Dec 2017 18:09:01 +0000"/>
                            <attachment id="172172" name="dd-rc8.tar" size="1000960" author="bruce.lucas@mongodb.com" created="Fri, 1 Dec 2017 18:08:49 +0000"/>
                            <attachment id="173594" name="dd.wt3767_68.zip" size="446463" author="neha.khatri" created="Fri, 15 Dec 2017 02:41:48 +0000"/>
                            <attachment id="173593" name="dd_mdb_master.zip" size="669446" author="neha.khatri" created="Fri, 15 Dec 2017 02:41:48 +0000"/>
                            <attachment id="180433" name="dd_wt3805.tgz" size="1072414" author="luke.chen@mongodb.com" created="Wed, 28 Feb 2018 03:32:43 +0000"/>
                            <attachment id="173474" name="master_vs_wt3767_3768.png" size="177138" author="neha.khatri" created="Thu, 14 Dec 2017 02:50:40 +0000"/>
                            <attachment id="180432" name="master_with_wt-3805.png" size="82706" author="luke.chen@mongodb.com" created="Wed, 28 Feb 2018 03:31:15 +0000"/>
                            <attachment id="172424" name="optimes.png" size="139662" author="bruce.lucas@mongodb.com" created="Tue, 5 Dec 2017 16:11:18 +0000"/>
                            <attachment id="172257" name="repro.sh" size="732" author="bruce.lucas@mongodb.com" created="Mon, 4 Dec 2017 13:30:22 +0000"/>
                            <attachment id="172230" name="smalloplog.png" size="136665" author="bruce.lucas@mongodb.com" created="Sun, 3 Dec 2017 15:04:52 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>18.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Dec 2017 04:28:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 50 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WT-3805'>WT-3805</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>julian.torres@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 50 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>luke.chen@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>neha.khatri</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htle5b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9de7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2125">Storage Non-NYC 2018-03-12</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htl093:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>