<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:23:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30238] High disk usage and query blocking on database drops</title>
                <link>https://jira.mongodb.org/browse/SERVER-30238</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This case is being raised from &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3207&quot; title=&quot;Drops with checkpoint_wait=false should not wait for checkpoints&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3207&quot;&gt;&lt;del&gt;WT-3207&lt;/del&gt;&lt;/a&gt; - in the latest 3.4.6 release, the bottleneck appears to have moved from CPU to disk.&lt;/p&gt;

&lt;p&gt;Image 1 - the database is dropped (09:02:47) and the queries, updates have dropped off.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/161023/161023_Starting.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Image 2 - Recovery - at 09:04:45 the primary recovers and starts processing requests. Note the list Databases that returned after a long period of time.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/161024/161024_recovered.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;A new run with diagnostics.data to follow - Can I have a secure link to submit it to?&lt;/p&gt;</description>
                <environment></environment>
        <key id="406322">SERVER-30238</key>
            <summary>High disk usage and query blocking on database drops</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="bruce.lucas@mongodb.com">Bruce Lucas</assignee>
                                    <reporter username="david.henderson@triggeredmessaging.com">David Henderson</reporter>
                        <labels>
                    </labels>
                <created>Thu, 20 Jul 2017 11:11:31 +0000</created>
                <updated>Thu, 24 Aug 2017 04:23:46 +0000</updated>
                            <resolved>Mon, 31 Jul 2017 14:45:41 +0000</resolved>
                                    <version>3.4.6</version>
                                                    <component>Performance</component>
                                        <votes>1</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1636273" author="richardyates" created="Mon, 31 Jul 2017 14:40:51 +0000"  >&lt;p&gt;Thanks Bruce, I agree it&apos;s best to start a new case for the specific issue you&apos;ve identified. Feel free to close this one down, Dave and I will keep an eye on the new ticket. Appreciate the workaround too, I&apos;ll give that a try.&lt;/p&gt;

&lt;p&gt;&amp;#8211; Richard.&lt;/p&gt;</comment>
                            <comment id="1636179" author="bruce.lucas@10gen.com" created="Mon, 31 Jul 2017 13:43:53 +0000"  >&lt;p&gt;Thanks Richard, that indeed has been helpful. The perf data has allowed us to pinpoint where we are spending the 30 seconds of CPU time that makes the dropDatabase operation slow in this case. I&apos;ve opened &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30426&quot; title=&quot;dropDatabase very slow due to repeated calls to closeAllCursors&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30426&quot;&gt;&lt;del&gt;SERVER-30426&lt;/del&gt;&lt;/a&gt; to focus on that specific problem as we have covered a couple of different issues in this ticket. Please follow that ticket for further updates.&lt;/p&gt;

&lt;p&gt;Meanwhile it is possible that you can mitigate the impact of this issue by dropping the individual collections in the database before dropping the database. The dropCollection command only takes a database lock, whereas the dropDatabase command takes a global lock, so the impact of the individual dropCollection commands will be limited primarily to the database that you are dropping.&lt;/p&gt;

&lt;p&gt;Thanks for bringing this issue to our attention. If you have no further questions I&apos;ll close this ticket out and we&apos;ll continue work on this specific issue on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30426&quot; title=&quot;dropDatabase very slow due to repeated calls to closeAllCursors&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30426&quot;&gt;&lt;del&gt;SERVER-30426&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Bruce&lt;/p&gt;
</comment>
                            <comment id="1636034" author="richardyates" created="Mon, 31 Jul 2017 11:07:27 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;Good news, I&apos;ve managed to reproduce easily with all the metrics collections going. I&apos;ve uploaded mongoanalysis.tar.gz which contains:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;A redacted loglevel 2 mongodb log file (many thanks for the pointer to the fruitsalad tool!)&lt;/li&gt;
	&lt;li&gt;Today&apos;s mongodb diag files&lt;/li&gt;
	&lt;li&gt;The unprocessed perf.data file + processed perfstacks.txt (as output by perf script in your previous comment)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;As requested I started the perf data collection, did a logrotate, turned on level 2 logging, left it for about 15 seconds then did the dropdatabase. I left the perf data going for the full 2 minutes and the dropdatabase + subsequent stall completed within that time.&lt;/p&gt;

&lt;p&gt;There won&apos;t be any other dropDatabase commands in that time so you should be able to see the exact start+completed times in the log file.&lt;/p&gt;

&lt;p&gt;Thanks for the continued support, hope that bundle helps with the investigation.&lt;br/&gt;
&amp;#8211; Richard.&lt;/p&gt;</comment>
                            <comment id="1632070" author="bruce.lucas@10gen.com" created="Wed, 26 Jul 2017 12:12:55 +0000"  >&lt;p&gt;Hi Richard,&lt;/p&gt;

&lt;p&gt;You can find a tool &lt;a href=&quot;https://github.com/rueckstiess/fruitsalad&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; that many of our customers have successfully used to redact log files. Let me know if you have any questions. &lt;/p&gt;

&lt;p&gt;Given that some work is required to reproduce the problem, I would suggest if possible capturing perf data at the same time. Some care is required as the volume of data capture by perf can be large. The following will capture perf data for 2 minutes, then process it into a form useful for our analysis:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;perf record  -a -g -F 99 sleep 120&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;perf script &amp;gt;perfstacks.txt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;The idea would be to start this, let it run for say 15-30 seconds to capture a baseline, then do the dropDatabase, then let perf continue to run to capture the stall and more baseline.&lt;/p&gt;

&lt;p&gt;If for any reason it&apos;s not practical to closely coordinate running perf with the dropDatabase, you can break up the perf data into one-minute segments with the following:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;while true; do perf record -a -g -F 99 -o perf.data.$(date -u +%FT%TZ) sleep 60; done&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Then after running dropDatabase, identify and process the data files that include the dropDatabase plus some surrounding data for baseline comparison:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for fn in ...; do perf script -i $fn &amp;gt;$fn.txt; done&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks you for your help.&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1632008" author="richardyates" created="Wed, 26 Jul 2017 09:28:06 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;I think that would be possible. My concern would be revealing our client&apos;s data in slow queries logged during that time but I will see if turning off slow query logging works without destroying the debug data you&apos;re after in the first place. Alternatively we&apos;ll have to see about obfuscating the data parts, if you have a handy tool for this (or some regex grep commands lying about!) that would be perfect.&lt;/p&gt;

&lt;p&gt;I&apos;m fresh out of databases to drop so give me a little time to get some data from backups to drop and I&apos;ll come back with my findings.&lt;/p&gt;</comment>
                            <comment id="1631167" author="bruce.lucas@10gen.com" created="Tue, 25 Jul 2017 15:04:56 +0000"  >&lt;p&gt;Hi Richard,&lt;/p&gt;

&lt;p&gt;The diagnostic data indicates that there is no longer a large amount of disk i/o, so removing the &quot;discard&quot; option has indeed addressed that issue.&lt;/p&gt;

&lt;p&gt;I&apos;ve spent some time trying to reproduce the remaining stall but so far have not succeeded. The diagnostic data indicates that ~300 WiredTiger tables (collections and indexes) were removed over the course of ~30 seconds, at an average rate of very close to 10 per second. This number makes me suspect possibly something is pacing the table drops at 10 per second. Unfortunately the stall has also created a gap in the diagnostic data, so we don&apos;t know in detail the timing of events during that 30 seconds.&lt;/p&gt;

&lt;p&gt;Would you be in a position to reproduce the problem while running at log level 2, and then upload the log file to the secure upload portal that we provided earlier?&lt;/p&gt;

&lt;p&gt;Beyond this we could also collect the following information while you reproduce the problem:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;perf data: this should have minimal impact on the system, but will only &quot;see&quot; CPU activity and won&apos;t see blocked threads. The diagnostic data indicates that there was significant CPU activity during the stall, so this may be sufficient. This can be set up to run for an extended period until the problem reproduces.&lt;/li&gt;
	&lt;li&gt;gdb stack traces: this will require timing the collection of the stack traces to coincide with a stall, but it will give more information as it will show what blocked threads are doing. It will block the mongod process for a few seconds, in addition to the stall.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Please let us know if either of the above options is possible (assuming the log file isn&apos;t sufficient) and we&apos;ll provide more details.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;
</comment>
                            <comment id="1629721" author="richardyates" created="Mon, 24 Jul 2017 08:02:09 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;Bad news I&apos;m afraid. After disabling discard on both primary+secondary nodes in a replicaset I delete another database around the same size as before (we do this on a weekly basis) and it still stalled the primary. The good news is it only stalled it for 30 seconds or so which is the quickest I&apos;ve seen so I think disabling discard has helped.&lt;/p&gt;

&lt;p&gt;I&apos;ve uploaded another metrics.interim file to the secure URL provided above if you&apos;re able to take another look?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Richard.&lt;/p&gt;</comment>
                            <comment id="1628691" author="bruce.lucas@10gen.com" created="Fri, 21 Jul 2017 15:48:22 +0000"  >&lt;p&gt;Hi Richard,&lt;/p&gt;

&lt;p&gt;We will indeed want to add this to our production notes once we have confirmation from you that it was the cause of your issue. I&apos;ll put this ticket in &quot;waiting for customer&quot; state pending that confirmation.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1628645" author="richardyates" created="Fri, 21 Jul 2017 15:26:11 +0000"  >&lt;p&gt;Hi Bruce,&lt;/p&gt;

&lt;p&gt;I&apos;m one of David&apos;s colleagues.&lt;/p&gt;

&lt;p&gt;Thanks for the pointers around discard + SSD TRIM. I&apos;ve done some more testing and forcing TRIM to run manually via fstrim caused identical behavior to what we&apos;ve been reporting so it definitely looks like you&apos;re onto something.&lt;/p&gt;

&lt;p&gt;We&apos;ll disable discard over the coming days (we have multiple replicasets so need a little time to demote primaries, remount etc) and let you know if that appears to be the cause.&lt;/p&gt;

&lt;p&gt;Since it was AWS&apos; recommendation to enable discard/TRIM support for instance volumes like we&apos;re using (although I can&apos;t find that reference now so they may have revised their advice!) can I suggest you amend your production notes to include specific mention of not using the discard mount option. We&apos;ll now plan to run fstrim manually periodically during a maintenance window to avoid any production impact and would probably recommend others do the same.&lt;/p&gt;

&lt;p&gt;David also found this relevant thread from mongodb-user: &lt;a href=&quot;https://groups.google.com/forum/#!topic/mongodb-user/Mj0x6m-02Ms&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/#!topic/mongodb-user/Mj0x6m-02Ms&lt;/a&gt; which sounds similar&lt;/p&gt;</comment>
                            <comment id="1628547" author="bruce.lucas@10gen.com" created="Fri, 21 Jul 2017 14:38:07 +0000"  >&lt;p&gt;Hi David,&lt;/p&gt;

&lt;p&gt;The diagnostic data that you uploaded showed about 31 GB of data being written to the device during the drop operation, which is about the compressed size of the db, that is its size on disk, as you reported.  However the metrics showed that the storage engine was not doing any i/o.&lt;/p&gt;

&lt;p&gt;I suspect that the &quot;discard&quot; mount option may be responsible for the i/o. This option &lt;a href=&quot;http://xfs.org/index.php/FITRIM/discard&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;causes&lt;/a&gt; trim commands to be issued to the device to discard unused blocks, and per the referenced article can have a severe performance impact. I believe this is causing the file removes involved in the dropDatabase operation to be very slow and generate a lot of i/o.&lt;/p&gt;

&lt;p&gt;Would it be possible to re-mount the filesystem without the &quot;discard&quot; option and test whether performance improves for dropDatabase?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Bruce&lt;/p&gt;</comment>
                            <comment id="1628501" author="david.henderson@triggeredmessaging.com" created="Fri, 21 Jul 2017 14:02:47 +0000"  >&lt;p&gt;/dev/nvme0n1 on /srv/mongodb type xfs (rw,noatime,discard)&lt;br/&gt;
On i3.2xlarge on AWS, Ubuntu 14.04.5 LTS, kernel: 4.4.0-78-generic&lt;/p&gt;</comment>
                            <comment id="1628488" author="bruce.lucas@10gen.com" created="Fri, 21 Jul 2017 13:53:57 +0000"  >&lt;p&gt;Thanks for that info David. Can you tell us what filesystem you are using and what mount options?&lt;/p&gt;</comment>
                            <comment id="1628443" author="david.henderson@triggeredmessaging.com" created="Fri, 21 Jul 2017 13:24:55 +0000"  >&lt;p&gt;It was ~ 26GB compressed around 107 collections, 2 indexes in each collection (_id, 1 compound index on _id, &quot;data.c&quot;)&lt;/p&gt;

&lt;p&gt;Estimated collection sizes:&lt;/p&gt;

&lt;p&gt;Biggest collection 26.9GB in size (uncompressed), 13.3 million docs&lt;br/&gt;
3 more collections between 1 and 2.3 million docs&lt;br/&gt;
then the rest under 1 million each&lt;/p&gt;</comment>
                            <comment id="1628440" author="bruce.lucas@10gen.com" created="Fri, 21 Jul 2017 13:15:34 +0000"  >&lt;p&gt;Thanks David. Can you tell us about the database being dropped - approximately how big is it (compressed size on disk if you know), and how many collections and indexes?&lt;/p&gt;</comment>
                            <comment id="1628259" author="david.henderson@triggeredmessaging.com" created="Fri, 21 Jul 2017 08:24:58 +0000"  >&lt;p&gt;Thanks - I&apos;ve uploaded the relevant files for a new instance of the issue. It seems that this issue is much easier to trigger than the previous version in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3207&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/WT-3207&lt;/a&gt; which tended to require multiple drops.&lt;/p&gt;</comment>
                            <comment id="1627734" author="thomas.schubert" created="Thu, 20 Jul 2017 17:33:09 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.henderson%40triggeredmessaging.com&quot; class=&quot;user-hover&quot; rel=&quot;david.henderson@triggeredmessaging.com&quot;&gt;david.henderson@triggeredmessaging.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/b36cc5d6-da46-4a3d-b357-9a3640f55741.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload for you to use&lt;/a&gt; to provide the diagnostic.data.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="410391">SERVER-30426</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 20 Jul 2017 17:33:09 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 28 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 28 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>david.henderson@triggeredmessaging.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>richardyates</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htbau7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht35kf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Drop database (WT, 3.4.6)&lt;/li&gt;
	&lt;li&gt;Queries and updates stall.&lt;/li&gt;
&lt;/ul&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htawwv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>