<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:55:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40703] Resource leakage + slowness after creating and deleting few thousands of DBs</title>
                <link>https://jira.mongodb.org/browse/SERVER-40703</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Creating new DB + collections + indexes and deleting it becomes slow after few thousands of cycles.&lt;/p&gt;

&lt;p&gt;The mongo process memory keeps growing although the DBs are deleted so only a single DB is present at any given time. At some point this cycle become more and more slower. It seems around the memory consumption of ~3.5GB.&lt;/p&gt;

&lt;p&gt;After stopping the script, the mongo process keeps consuming large amount of CPU and Memory (forever).&lt;br/&gt;
If script will be run again, memory will get even higher and also the cycle time.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;See a simple script to reproduce the issue:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/test.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/test.js&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;And results:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/3.6.11-in-memory.png&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/3.6.11-in-memory.png&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/3.6.11-regular.png&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/assaf-xm/mongodb-create-drop-dbs/blob/master/3.6.11-regular.png&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;At start timings are fast - 201ms:&lt;/p&gt;

&lt;p&gt;2019-04-17T12:11:25.645Z start 3&lt;br/&gt;
 2019-04-17T12:11:25.654Z connected 3&lt;br/&gt;
 2019-04-17T12:11:25.655Z collection 0&lt;br/&gt;
 2019-04-17T12:11:25.672Z collection 1&lt;br/&gt;
 2019-04-17T12:11:25.690Z collection 2&lt;br/&gt;
 2019-04-17T12:11:25.706Z collection 3&lt;br/&gt;
 2019-04-17T12:11:25.726Z collection 4&lt;br/&gt;
 2019-04-17T12:11:25.742Z collection 5&lt;br/&gt;
 2019-04-17T12:11:25.760Z collection 6&lt;br/&gt;
 2019-04-17T12:11:25.779Z collection 7&lt;br/&gt;
 2019-04-17T12:11:25.796Z collection 8&lt;br/&gt;
 2019-04-17T12:11:25.812Z collection 9&lt;br/&gt;
 2019-04-17T12:11:25.828Z drop collection 0&lt;br/&gt;
 2019-04-17T12:11:25.830Z drop collection 1&lt;br/&gt;
 2019-04-17T12:11:25.832Z drop collection 2&lt;br/&gt;
 2019-04-17T12:11:25.833Z drop collection 3&lt;br/&gt;
 2019-04-17T12:11:25.835Z drop collection 4&lt;br/&gt;
 2019-04-17T12:11:25.836Z drop collection 5&lt;br/&gt;
 2019-04-17T12:11:25.838Z drop collection 6&lt;br/&gt;
 2019-04-17T12:11:25.840Z drop collection 7&lt;br/&gt;
 2019-04-17T12:11:25.841Z drop collection 8&lt;br/&gt;
 2019-04-17T12:11:25.843Z drop collection 9&lt;br/&gt;
 2019-04-17T12:11:25.845Z drop 3&lt;br/&gt;
 2019-04-17T12:11:25.846Z finish 3 | 201&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;But after 8800 cycles DB operations are much slower - 685ms:&lt;/p&gt;

&lt;p&gt;2019-04-17T12:41:20.397Z start 8800&lt;br/&gt;
 2019-04-17T12:41:20.402Z connected 8800&lt;br/&gt;
 2019-04-17T12:41:20.402Z collection 0&lt;br/&gt;
 2019-04-17T12:41:20.486Z collection 1&lt;br/&gt;
 2019-04-17T12:41:20.557Z collection 2&lt;br/&gt;
 2019-04-17T12:41:20.642Z collection 3&lt;br/&gt;
 2019-04-17T12:41:20.716Z collection 4&lt;br/&gt;
 2019-04-17T12:41:20.773Z collection 5&lt;br/&gt;
 2019-04-17T12:41:20.831Z collection 6&lt;br/&gt;
 2019-04-17T12:41:20.877Z collection 7&lt;br/&gt;
 2019-04-17T12:41:20.916Z collection 8&lt;br/&gt;
 2019-04-17T12:41:20.959Z collection 9&lt;br/&gt;
 2019-04-17T12:41:20.999Z drop collection 0&lt;br/&gt;
 2019-04-17T12:41:21.003Z drop collection 1&lt;br/&gt;
 2019-04-17T12:41:21.009Z drop collection 2&lt;br/&gt;
 2019-04-17T12:41:21.018Z drop collection 3&lt;br/&gt;
 2019-04-17T12:41:21.027Z drop collection 4&lt;br/&gt;
 2019-04-17T12:41:21.036Z drop collection 5&lt;br/&gt;
 2019-04-17T12:41:21.042Z drop collection 6&lt;br/&gt;
 2019-04-17T12:41:21.052Z drop collection 7&lt;br/&gt;
 2019-04-17T12:41:21.061Z drop collection 8&lt;br/&gt;
 2019-04-17T12:41:21.071Z drop collection 9&lt;br/&gt;
 2019-04-17T12:41:21.080Z drop 8800&lt;br/&gt;
 2019-04-17T12:41:21.082Z finish 8800 | 685&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment>Windows &amp;amp; Linux</environment>
        <key id="740629">SERVER-40703</key>
            <summary>Resource leakage + slowness after creating and deleting few thousands of DBs</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-storage-engines">Backlog - Storage Engines Team</assignee>
                                    <reporter username="assaf-oren">assaf-oren</reporter>
                        <labels>
                    </labels>
                <created>Wed, 17 Apr 2019 21:11:42 +0000</created>
                <updated>Tue, 6 Dec 2022 03:01:29 +0000</updated>
                            <resolved>Fri, 26 Jun 2020 04:13:32 +0000</resolved>
                                    <version>3.6.11</version>
                                                                        <votes>1</votes>
                                    <watches>17</watches>
                                                                                                                <comments>
                            <comment id="3225155" author="brian.lane" created="Fri, 26 Jun 2020 04:12:48 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=assaf%40xmcyber.com&quot; class=&quot;user-hover&quot; rel=&quot;assaf@xmcyber.com&quot;&gt;assaf@xmcyber.com&lt;/a&gt; - great to hear.&lt;/p&gt;

&lt;p&gt;-Brian&lt;/p&gt;</comment>
                            <comment id="3223331" author="assaf-oren" created="Thu, 25 Jun 2020 08:33:47 +0000"  >&lt;p&gt;Following our latest conversation with Kian Gogan, I checked this with mongo 4.2.3 and the issue doesn&apos;t seem to reproduce.&lt;/p&gt;

&lt;p&gt;Therefore I think we can close this ticket.&lt;/p&gt;</comment>
                            <comment id="2880631" author="assaf-oren" created="Mon, 17 Feb 2020 07:19:05 +0000"  >&lt;p&gt;I can see that this fix was delayed.&lt;/p&gt;

&lt;p&gt;Is there any workaround that can be done using commands to prevent restarting the DB when the memory gets high (and performance gets low)?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2224967" author="bruce.lucas@10gen.com" created="Thu, 25 Apr 2019 17:33:48 +0000"  >&lt;p&gt;The issue reproduces on 4.1.10, and with WT (not just in-memory). Here&apos;s a run on 4.1.10, standalone, WT, cache size 0.2 GB, with the heap profiler enabled:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/215190/215190_heap.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;When the cache reached 95% the &quot;bytes not belonging to page images&quot; were finally freed (evicted?), and based on the behavior of the heap profile these seem like likely contributors:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-04-25T16:53:55.096+0000 I  -        [ftdc] heapProfile stack307: { 0: &quot;tc_calloc&quot;, 1: &quot;__wt_calloc&quot;, 2: &quot;__wt_update_alloc&quot;, 3: &quot;__wt_row_modify&quot;, 4: &quot;__wt_btcur_insert&quot;, 5: &quot;0x5616119bbfef&quot;, 6: &quot;__wt_metadata_update&quot;, 7: &quot;0x5616119f363f&quot;, 8: &quot;__wt_meta_ckptlist_set&quot;, 9: &quot;0x561611948791&quot;, 10: &quot;__wt_checkpoint_close&quot;, 11: &quot;__wt_conn_dhandle_close&quot;, 12: &quot;__wt_session_release_dhandle&quot;, 13: &quot;0x5616119bd668&quot;, 14: &quot;mongo::WiredTigerIndex::StandardBulkBuilder::~StandardBulkBuilder&quot;, 15: &quot;mongo::AbstractIndexAccessMethod::commitBulk&quot;, 16: &quot;mongo::MultiIndexBlock::dumpInsertsFromBulk&quot;, 17: &quot;mongo::MultiIndexBlock::insertAllDocumentsInCollection&quot;, 18: &quot;0x561611d48659&quot;, 19: &quot;mongo::ErrmsgCommandDeprecated::run&quot;, 20: &quot;mongo::BasicCommand::Invocation::run&quot;, 21: &quot;0x561611a94d15&quot;, 22: &quot;0x561611a9637c&quot;, 23: &quot;0x561611a971fe&quot;, 24: &quot;mongo::ServiceEntryPointCommon::handleRequest&quot;, 25: &quot;mongo::ServiceEntryPointMongod::handleRequest&quot;, 26: &quot;mongo::ServiceStateMachine::_processMessage&quot;, 27: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 28: &quot;0x561611a9059d&quot;, 29: &quot;mongo::transport::ServiceExecutorSynchronous::schedule&quot;, 30: &quot;mongo::ServiceStateMachine::_scheduleNextWithGuard&quot;, 31: &quot;mongo::ServiceStateMachine::_sourceCallback&quot;, 32: &quot;mongo::ServiceStateMachine::_sourceMessage&quot;, 33: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 34: &quot;0x561611a9059d&quot;, 35: &quot;0x56161294253b&quot;, 36: &quot;0x561612f6d3d4&quot;, 37: &quot;0x7f1c25cb8dd5&quot;, 38: &quot;clone&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-04-25T16:53:53.227+0000 I  -        [ftdc] heapProfile stack245: { 0: &quot;tc_calloc&quot;, 1: &quot;__wt_calloc&quot;, 2: &quot;__wt_update_alloc&quot;, 3: &quot;__wt_row_modify&quot;, 4: &quot;__wt_btcur_insert&quot;, 5: &quot;0x5616119bbfef&quot;, 6: &quot;__wt_metadata_insert&quot;, 7: &quot;__wt_schema_create&quot;, 8: &quot;0x56161191e930&quot;, 9: &quot;__wt_schema_create&quot;, 10: &quot;__wt_session_create&quot;, 11: &quot;0x561611931a0c&quot;, 12: &quot;mongo::WiredTigerKVEngine::makeTemporaryRecordStore&quot;, 13: &quot;mongo::KVStorageEngine::makeTemporaryRecordStore&quot;, 14: &quot;mongo::IndexBuildInterceptor::IndexBuildInterceptor&quot;, 15: &quot;mongo::IndexCatalogImpl::IndexBuildBlock::init&quot;, 16: &quot;mongo::MultiIndexBlock::init&quot;, 17: &quot;0x561611d44d2a&quot;, 18: &quot;0x561611d48d40&quot;, 19: &quot;mongo::ErrmsgCommandDeprecated::run&quot;, 20: &quot;mongo::BasicCommand::Invocation::run&quot;, 21: &quot;0x561611a94d15&quot;, 22: &quot;0x561611a9637c&quot;, 23: &quot;0x561611a971fe&quot;, 24: &quot;mongo::ServiceEntryPointCommon::handleRequest&quot;, 25: &quot;mongo::ServiceEntryPointMongod::handleRequest&quot;, 26: &quot;mongo::ServiceStateMachine::_processMessage&quot;, 27: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 28: &quot;0x561611a9059d&quot;, 29: &quot;mongo::transport::ServiceExecutorSynchronous::schedule&quot;, 30: &quot;mongo::ServiceStateMachine::_scheduleNextWithGuard&quot;, 31: &quot;mongo::ServiceStateMachine::_sourceCallback&quot;, 32: &quot;mongo::ServiceStateMachine::_sourceMessage&quot;, 33: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 34: &quot;0x561611a9059d&quot;, 35: &quot;0x56161294253b&quot;, 36: &quot;0x561612f6d3d4&quot;, 37: &quot;0x7f1c25cb8dd5&quot;, 38: &quot;clone&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2019-04-25T16:53:53.122+0000 I  -        [ftdc] heapProfile stack206: { 0: &quot;tc_calloc&quot;, 1: &quot;__wt_calloc&quot;, 2: &quot;__wt_update_alloc&quot;, 3: &quot;__wt_row_modify&quot;, 4: &quot;__wt_btcur_insert&quot;, 5: &quot;0x5616119bbfef&quot;, 6: &quot;__wt_metadata_insert&quot;, 7: &quot;__wt_schema_create&quot;, 8: &quot;0x56161191e930&quot;, 9: &quot;__wt_schema_create&quot;, 10: &quot;__wt_session_create&quot;, 11: &quot;0x561611931a0c&quot;, 12: &quot;mongo::WiredTigerIndex::Create&quot;, 13: &quot;mongo::WiredTigerKVEngine::createGroupedSortedDataInterface&quot;, 14: &quot;mongo::KVCollectionCatalogEntry::prepareForIndexBuild&quot;, 15: &quot;mongo::IndexCatalogImpl::IndexBuildBlock::init&quot;, 16: &quot;mongo::IndexCatalogImpl::createIndexOnEmptyCollection&quot;, 17: &quot;mongo::DatabaseImpl::createCollection&quot;, 18: &quot;0x561611d449c3&quot;, 19: &quot;0x561611d48c81&quot;, 20: &quot;mongo::ErrmsgCommandDeprecated::run&quot;, 21: &quot;mongo::BasicCommand::Invocation::run&quot;, 22: &quot;0x561611a94d15&quot;, 23: &quot;0x561611a9637c&quot;, 24: &quot;0x561611a971fe&quot;, 25: &quot;mongo::ServiceEntryPointCommon::handleRequest&quot;, 26: &quot;mongo::ServiceEntryPointMongod::handleRequest&quot;, 27: &quot;mongo::ServiceStateMachine::_processMessage&quot;, 28: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 29: &quot;0x561611a9059d&quot;, 30: &quot;mongo::transport::ServiceExecutorSynchronous::schedule&quot;, 31: &quot;mongo::ServiceStateMachine::_scheduleNextWithGuard&quot;, 32: &quot;mongo::ServiceStateMachine::_sourceCallback&quot;, 33: &quot;mongo::ServiceStateMachine::_sourceMessage&quot;, 34: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 35: &quot;0x561611a9059d&quot;, 36: &quot;0x56161294253b&quot;, 37: &quot;0x561612f6d3d4&quot;, 38: &quot;0x7f1c25cb8dd5&quot;, 39: &quot;clone&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;This event however did not delete data handles, and application performance impact due to &quot;dhandle lock application thread time waiting&quot; continued growing.&lt;/p&gt;</comment>
                            <comment id="2224877" author="bruce.lucas@10gen.com" created="Thu, 25 Apr 2019 16:33:48 +0000"  >&lt;p&gt;Simple shell-based repro:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ncolls = 10&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;niter = 2000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;function repro() {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    for (let i = 0; i &amp;lt; niter; i++) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        if (i%100==0)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            print(&apos;iteration&apos;, i)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (let j = 0; j &amp;lt; ncolls; j++) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c = db[&apos;c&apos;+j]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({a: 1, b: 1, c: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({b: 1, a: 1, c: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({c: 1, b: 1, a: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({a: 1, c: 1, b: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({b: 1, c: 1, a: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.createIndex({c: 1, a: 1, b: 1});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.insert({a: 1, b: 2, c: 3});&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        for (let j = 0; j &amp;lt; ncolls; j++) {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c = db[&apos;c&apos;+j]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;            c.drop()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;        }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Following run uses the in-memory storage engine, 3.6.11, running standalone, with a cache size (&lt;tt&gt;--inMemorySizeGB&lt;/tt&gt;) of 1 GB.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/215179/215179_dhandles.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Even though the application drops all collections that it creates, we see constant growth in &quot;data handles currently active&quot; and &quot;bytes not belonging to page images in the cache&quot;&lt;/li&gt;
	&lt;li&gt;Table create and table drop calls closely match, so I think the problem is in the storage engine, not the integration layer.&lt;/li&gt;
	&lt;li&gt;At A cache reaches 5% dirty (due to &quot;bytes not belonging to page images in the cache&quot;), evictions begin, and performance drops. This is accounted for by &quot;dhandle lock application thread time waiting&quot;.&lt;/li&gt;
	&lt;li&gt;After evictions begin at A the profile shows ~1 thread constantly active in __evict_walk @evict_lru.c:1432 (highlighted in blue), which is this line of code:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		/* Ignore non-btree handles, or handles that aren&apos;t open. */&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		if (dhandle-&amp;gt;type != WT_DHANDLE_TYPE_BTREE ||&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		    !F_ISSET(dhandle, WT_DHANDLE_OPEN))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;			continue;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Does this mean that the eviction server is spending a lot of its time skipping over handles that aren&apos;t open?&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Shortly after A &quot;data handles currently active&quot; begins to rise&lt;/li&gt;
	&lt;li&gt;Data handles and bytes not belonging to page images never goes down, even after workload ends.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2219838" author="assaf-oren" created="Sat, 20 Apr 2019 06:15:31 +0000"  >&lt;p&gt;Also uploaded the diagnostic data of the last few days to the secure upload portal.&#160;&lt;/p&gt;</comment>
                            <comment id="2219837" author="assaf-oren" created="Sat, 20 Apr 2019 06:03:24 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I did one more run of the script letting it run for hours (30000 cycles). Used disk storage. See the file &apos;3.6.11-30000_cycles-disk_storage-WiredTiger.png&apos;&lt;/p&gt;

&lt;p&gt;As you can see the latency starts to grow after ~14000 cycles, climbing up to 1100ms, but it drop from time to time back to ~200ms.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Assaf&lt;/p&gt;</comment>
                            <comment id="2219062" author="eric.sedor" created="Fri, 19 Apr 2019 14:18:29 +0000"  >&lt;p&gt;Yes Assaf, you can use this secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/f35ec4dd-9085-4906-8c83-f25733494a6a.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt;. Files uploaded here are visible only to MongoDB employees and deleted after some time.&lt;/p&gt;</comment>
                            <comment id="2218789" author="assaf-oren" created="Fri, 19 Apr 2019 06:10:25 +0000"  >&lt;p&gt;Hi Eric,&lt;/p&gt;

&lt;p&gt;For your question, the memory growth was seen both for the in-memory and disk storage engines.&lt;/p&gt;

&lt;p&gt;In the in-memory it is faster to reproduce the point in which a cycle (creating and delete DB and collections) latency starts to grow as well. When it starts to grow it continue to grow as you can see from the graphs and DB CPU will continue to be high even when the script is stopped.&lt;/p&gt;

&lt;p&gt;(for the in-memory the creation and deletion of DBs are ~7 times faster, so the issue can be faster reproduced)&lt;/p&gt;

&lt;p&gt;I will run the script again and provide you with the diagnostic.data, can you provide a secured private location for these files?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Assaf&#160;&lt;/p&gt;</comment>
                            <comment id="2218492" author="eric.sedor" created="Thu, 18 Apr 2019 20:50:51 +0000"  >&lt;p&gt;Thanks for your report &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=assaf-oren&quot; class=&quot;user-hover&quot; rel=&quot;assaf-oren&quot;&gt;assaf-oren&lt;/a&gt;. Can you clarify: Are you reporting this unexpected memory growth using the in-memory storage engine?&lt;/p&gt;

&lt;p&gt;We are testing your script internally but it will also be helpful if you can archive (tar or zip) the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory for your test cluster (described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and attach it to this ticket.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1351126">SERVER-48213</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="751510">SERVER-40925</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="661710">WT-4513</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1351126">SERVER-48213</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="214716" name="3.6.11-30000_cycles-disk_storage-WiredTiger.png" size="105214" author="assaf-oren" created="Sat, 20 Apr 2019 05:56:50 +0000"/>
                            <attachment id="214698" name="Screen Shot 2019-04-18 at 4.39.37 PM.png" size="61359" author="eric.sedor@mongodb.com" created="Fri, 19 Apr 2019 20:00:29 +0000"/>
                            <attachment id="215179" name="dhandles.png" size="519423" author="bruce.lucas@mongodb.com" created="Thu, 25 Apr 2019 16:31:37 +0000"/>
                            <attachment id="215190" name="heap.png" size="236881" author="bruce.lucas@mongodb.com" created="Thu, 25 Apr 2019 17:32:53 +0000"/>
                            <attachment id="214928" name="test.js" size="2866" author="eric.sedor@mongodb.com" created="Tue, 23 Apr 2019 19:36:37 +0000"/>
                            <attachment id="215193" name="x-4110.tgz" size="8179575" author="bruce.lucas@mongodb.com" created="Thu, 25 Apr 2019 17:43:32 +0000"/>
                            <attachment id="215192" name="x-in-memory.tgz" size="123117" author="bruce.lucas@mongodb.com" created="Thu, 25 Apr 2019 17:43:03 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25135"><![CDATA[Storage Engines]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 17 Apr 2019 23:52:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 32 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 32 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>assaf-oren</customfieldvalue>
            <customfieldvalue>backlog-server-storage-engines</customfieldvalue>
            <customfieldvalue>brian.lane@mongodb.com</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huuuu7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr71tj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10555" key="com.atlassian.jira.plugin.system.customfieldtypes:float">
                        <customfieldname>Story Points</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
        <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huuh3j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>