<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:21:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-50014] Mongos can accumulate ChunkManager of dropped collections no longer being used</title>
                <link>https://jira.mongodb.org/browse/SERVER-50014</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Mongos don&apos;t proactively clear the cached metadata unless it was the mongos that dropped it (or when it tries to perform an operation on a namespace that was already dropped), so cached metadata for dropped collections can stay in the cache forever. This can be problematic for use cases where sharded collections are constantly being created and dropped, as they in theory should net no additional memory required, but will cause mongos to slowly accumulate chunk metadata for old dropped collections.&lt;/p&gt;

&lt;p&gt;Original title: mongos memory footprint keeps increasing&lt;/p&gt;

&lt;p&gt;Original description&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Machine type: Amazon Linux 2&lt;/p&gt;

&lt;p&gt;Shard count: 5&lt;/p&gt;

&lt;p&gt;Cluster Architecture: PSA&lt;/p&gt;

&lt;p&gt;Config nodes: 3&lt;/p&gt;

&lt;p&gt;Mongo Version: 4.2.6&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;We have a mongos running on each application server instance. The mongos seems to continuously keep on accumulating memory overtime until the mongos service is restarted.&#160;&lt;/p&gt;

&lt;p&gt;I don&apos;t see a mongos configuration option (&lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongos/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/reference/program/mongos/&lt;/a&gt;) to control the memory growth beyond a certain threshold. Can it be configurable like wiredTigerCacheSize?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;This seems like a memory leak issue! Is there a way we can inspect what kind of mongos related data is stored in memory?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Attaching the serverStatus output..&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;&lt;/blockquote&gt;</description>
                <environment></environment>
        <key id="1424382">SERVER-50014</key>
            <summary>Mongos can accumulate ChunkManager of dropped collections no longer being used</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="stephenpaul2727@gmail.com">Stephen Paul Adithela</reporter>
                        <labels>
                    </labels>
                <created>Thu, 30 Jul 2020 00:25:51 +0000</created>
                <updated>Fri, 27 Oct 2023 13:52:45 +0000</updated>
                            <resolved>Fri, 8 Jan 2021 07:01:31 +0000</resolved>
                                    <version>4.2.6</version>
                                                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="3554399" author="kaloian.manassiev" created="Fri, 8 Jan 2021 07:01:13 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed report and for suggesting the workaround, and apologies for the delayed response!&lt;/p&gt;

&lt;p&gt;In the latest (not yet released version of MongoDB) we did some work to address this problem, by capping the cache of routing information that the routers keep and making that &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a2bd60fb04358ce7250c87477365a26b660b907a/src/mongo/s/catalog_cache.cpp#L64&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;cache an LRU&lt;/a&gt; (&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46199&quot; title=&quot;Implement collection cache on top of ReadThroughCache to make it causally consistent&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46199&quot;&gt;&lt;del&gt;SERVER-46199&lt;/del&gt;&lt;/a&gt;). For collections with large number of chunks, this still doesn&apos;t probably help, because 10,000 is a large number, but at least there is a cap on the number of collections we will cache.&lt;/p&gt;

&lt;p&gt;We have some plans underway to make the memory accounting of these caches better, but it might take some time to implement. In the mean time, I am closing this ticket with the understanding that the workaround is to run the &lt;tt&gt;flushRouterConfig&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;



</comment>
                            <comment id="3403217" author="renctan" created="Mon, 21 Sep 2020 15:08:47 +0000"  >&lt;p&gt;I think the workaround should work.&lt;/p&gt;</comment>
                            <comment id="3401665" author="stephenpaul2727@gmail.com" created="Fri, 18 Sep 2020 20:00:33 +0000"  >&lt;p&gt;Thanks! If I were to create a workaround script to do a count query on all &lt;b&gt;dropped:true&lt;/b&gt;&#160;collections in config DB (from all mongos&apos;s), this should force mongos to access the ns and hence could clear the metadata of dropped collection right? I am trying to avoid restarting mongos and wondering you think this could be possible?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3400984" author="renctan" created="Fri, 18 Sep 2020 15:13:44 +0000"  >&lt;p&gt;They currently do. Dropped collections stay in config.collections with dropped: true. However, the mongos currently doesn&apos;t actively check that the collection is already dropped. They will only learn about it when it attempts to access the namespace and the shard responds back that the collection is dropped.&lt;/p&gt;</comment>
                            <comment id="3399973" author="stephenpaul2727@gmail.com" created="Thu, 17 Sep 2020 20:59:42 +0000"  >&lt;p&gt;Thanks Randolph, Yes restarting mongos will definitely release all that cached memory. Could the mongo config servers store this drop sharded collection info and then be propogated to rest of mongos nodes?&lt;/p&gt;</comment>
                            <comment id="3399939" author="renctan" created="Thu, 17 Sep 2020 20:40:19 +0000"  >&lt;p&gt;If all the mongos accessed the sharded collection in the past, they will have the metadata cached. When the drop is called, only the specific mongos that ran the drop will clear the cache for that collection. The rest of the mongos will be oblivious that the collection has been dropped. Unfortunately, we currently don&apos;t have a convenient way to tell these mongos to purge cached metadata for collections that were already dropped, so I&apos;m hesitant to provide a complicated procedure. The only sure fire way to free the memory is to restart the mongos.&lt;/p&gt;</comment>
                            <comment id="3399815" author="stephenpaul2727@gmail.com" created="Thu, 17 Sep 2020 19:26:11 +0000"  >&lt;p&gt;Thanks randolph for the info. It makes sense. Our java app (using mongo java drivers) is currently dropping and recreating these sharded collections everyday. If what i understand is correct, you are saying if the drop and recreate is directly done on mongo CLI, then cached metadata should get cleaned up right?&lt;/p&gt;



&lt;p&gt;Would this cached metadata for chunkManager get distributed across all mongos VM&apos;s memory? For example, if we have 10 mongos&apos;s running on separate VM&apos;s and drop sharded col is issued on all mongos&apos;s (only one mongos got success &amp;amp; rest say already dropped), Does this cached metadata for dropped sharded col only get stored on succeeded mongos VM memory or all the mongos&apos;s VM&apos;s memory?&lt;/p&gt;


&lt;p&gt;Thank you!&lt;/p&gt;</comment>
                            <comment id="3399785" author="renctan" created="Thu, 17 Sep 2020 19:08:30 +0000"  >&lt;p&gt;I see. Currently, mongos don&apos;t proactively cleanup cached metadata for collections that are dropped (unless it was the mongos were the drop was called) or when it tries to perform an operation on that namespace and discovers that it is already dropped. This would explain the slow accumulation of RoutingTable::makeNew objects not being freed.&lt;/p&gt;</comment>
                            <comment id="3399712" author="stephenpaul2727@gmail.com" created="Thu, 17 Sep 2020 18:31:28 +0000"  >&lt;p&gt;Hi Randolph,&lt;br/&gt;
Yes, everyday we have 3-4 sharded collections being created and 3-4 sharded collections getting dropped. Currently, all mongos&apos;s are experiencing this constant memory growth pattern&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3399661" author="renctan" created="Thu, 17 Sep 2020 18:11:11 +0000"  >&lt;p&gt;Are sharded collections regularly being created and dropped in this cluster? How many mongos are experiencing this issue vs mongos not experiencing this issue?&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</comment>
                            <comment id="3395566" author="stephenpaul2727@gmail.com" created="Tue, 15 Sep 2020 21:26:52 +0000"  >&lt;p&gt;Hi Dima, Attaching config.chunks stats output&#160;&lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/278526/278526_chunks-col-stats.txt&quot; title=&quot;chunks-col-stats.txt attached to SERVER-50014&quot;&gt;chunks-col-stats.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="3394166" author="dmitry.agranat" created="Tue, 15 Sep 2020 12:14:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, while we are still investigating, could you provide the size of your &lt;tt&gt;config.chunks&lt;/tt&gt; collection?&lt;/p&gt;</comment>
                            <comment id="3391544" author="stephenpaul2727@gmail.com" created="Mon, 14 Sep 2020 06:41:52 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;We have 1054829 chunks in our cluster at the moment.&#160;  &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/278176/278176_image-2020-09-13-23-41-29-108.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Distribution per shard:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/278177/278177_image-2020-09-13-23-48-11-445.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                            <comment id="3391242" author="dmitry.agranat" created="Sun, 13 Sep 2020 13:26:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, please hold off with setting &lt;tt&gt;tcmallocAggressiveMemoryDecommit&lt;/tt&gt; as proposed in my last comment, we are still investigating. Could you also clarify how many chunks does this cluster have?&lt;/p&gt;</comment>
                            <comment id="3391210" author="dmitry.agranat" created="Sun, 13 Sep 2020 09:44:46 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, thanks for uploading the requested information.&lt;/p&gt;

&lt;p&gt;Unfortunately, some of the stacks which are present in our &lt;tt&gt;diagnostic.data&lt;/tt&gt;, are not present in the logs you&apos;ve attached (grep output).&lt;/p&gt;

&lt;p&gt;I did manage to catch one stack which is related to getMore commands and this might indicate &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36346&quot; title=&quot;Large memory consumption per pending getmore&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36346&quot;&gt;&lt;del&gt;SERVER-36346&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos.log.2020-08-27T00-00-01:2020-08-26T03:00:43.021+0000 I  -        [ftdc] heapProfile stack881: { 0: &quot;realloc&quot;, 1: &quot;mongo::mongoRealloc&quot;, 2: &quot;mongo::_BufBuilder&amp;lt;mongo::SharedBufferAllocator&amp;gt;::grow_reallocate&quot;, 3: &quot;mongo::replyToQuery&quot;, 4: &quot;mongo::Strategy::getMore&quot;, 5: &quot;mongo::ServiceEntryPointMongos::handleRequest&quot;, 6: &quot;mongo::ServiceStateMachine::_processMessage&quot;, 7: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 8: &quot;0x56209755032c&quot;, 9: &quot;mongo::transport::ServiceExecutorSynchronous::schedule&quot;, 10: &quot;mongo::ServiceStateMachine::_scheduleNextWithGuard&quot;, 11: &quot;mongo::ServiceStateMachine::_sourceCallback&quot;, 12: &quot;mongo::ServiceStateMachine::_sourceMessage&quot;, 13: &quot;mongo::ServiceStateMachine::_runNextInGuard&quot;, 14: &quot;0x56209755032c&quot;, 15: &quot;0x5620978d7a1b&quot;, 16: &quot;0x562097fea785&quot;, 17: &quot;0x562097fea7e4&quot;, 18: &quot;0x7ff704a6f40b&quot;, 19: &quot;clone&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;However, given the incomplete logs, I was not able to collect enough stacks which would account for the amount of memory growth we are looking for. Another issue that I saw is a slight memory fragmentation but given a very small memory fluctuations for this workload, it will be rather difficult to determine the cause here. &lt;/p&gt;

&lt;p&gt;Apart for the mentioned possible cause as &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-36346&quot; title=&quot;Large memory consumption per pending getmore&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-36346&quot;&gt;&lt;del&gt;SERVER-36346&lt;/del&gt;&lt;/a&gt;, I&apos;d like to see if de-commiting memory more frequently would help in this situation by setting:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({setParameter: 1, tcmallocAggressiveMemoryDecommit: 1})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3377697" author="stephenpaul2727@gmail.com" created="Mon, 7 Sep 2020 21:03:36 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Yes, logs are also uploaded to that same secure portal with heapProfile grepped&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;</comment>
                            <comment id="3376744" author="dmitry.agranat" created="Sun, 6 Sep 2020 10:14:34 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, could you point me to the data requested in step 6:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;From the mongoS logs, extract everything with heapProfile stack and upload it to the same secure portal&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;If those logs are still available, you can just grep for &lt;tt&gt;heapProfile&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3375805" author="stephenpaul2727@gmail.com" created="Thu, 3 Sep 2020 22:49:42 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I have uploaded all the files on secure portal according to the instructions in the previous comment.&lt;br/&gt;
Suffix tt --&amp;gt; problematic ones where mongos memory issues are seen&lt;/p&gt;

&lt;p&gt;Suffix lla --&amp;gt; non-problematic&lt;/p&gt;</comment>
                            <comment id="3373953" author="dmitry.agranat" created="Thu, 3 Sep 2020 10:37:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, did you have a chance to review my last comment?&lt;/p&gt;</comment>
                            <comment id="3353822" author="dmitry.agranat" created="Sun, 23 Aug 2020 06:46:54 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for clarification.&lt;/p&gt;

&lt;p&gt;I suggest we&apos;ll re-collect the requested data but with some modification.&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Increase the &lt;tt&gt;diagnostic.data&lt;/tt&gt; collection from the default 200 MB to 1000 MB by modifying this parameter: &lt;a href=&quot;https://docs.mongodb.com/manual/reference/parameters/index.html#param.diagnosticDataCollectionDirectorySizeMB&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;diagnosticDataCollectionDirectorySizeMB&lt;/a&gt;. This will allow us to have a longer time window we can look back and see the memory utilization. This is required due to a relatively small memory growth and the need to collect data for, perhaps, for more than a week&lt;/li&gt;
	&lt;li&gt;Make sure log retention is more than a week, let&apos;s say, 10 days&lt;/li&gt;
	&lt;li&gt;Restart mongoS with &lt;tt&gt;heapProfilingEnabled=true&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;Let mongoS run for a week&lt;/li&gt;
	&lt;li&gt;Save the &lt;tt&gt;diagnostic.data&lt;/tt&gt; and mongoS server logs&lt;/li&gt;
	&lt;li&gt;Upload the &lt;tt&gt;diagnostic.data&lt;/tt&gt; to provided secure portal. From the mongoS logs, extract everything with &lt;b&gt;heapProfile stack&lt;/b&gt; and upload it to the same secure portal&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3351630" author="stephenpaul2727@gmail.com" created="Fri, 21 Aug 2020 04:16:54 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;Thanks for the reply! our current VM (4core/8GB) has two services running. One is a java application (which takes 60% of total mem) at any time. With GC, the memory upper limit more or less stays the same. mongos starts with less amount of memory but keeps continuously growing. as you have noticed 50MB per day. with 60% already occupied, we can only afford another 20-30% for mongos. But it grows past that point which invokes OOM Killer.&lt;/p&gt;

&lt;p&gt;We have the same mongos version (4.2.6) running in another environment (with same java process, same VM specs and same IO rate),&#160; In that environment, mongos memory growth is much more stable.&#160;&lt;/p&gt;

&lt;p&gt;On the problematic VM, there is a growth of 5% over the past &lt;b&gt;7 days&lt;/b&gt;. so ~410MB. On the other similar environment, the mongos growth is less than 1-2% for the past &lt;b&gt;30 days&lt;/b&gt;. Attaching the screenshot of memory growth on problematic VM &amp;amp; Normal VM&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I understand delving deep into this issue might be hard online. I am okay to enable heap profiling again on both these VMs, collect logs and observe for a few days. Please let me know your thoughts. Also open to any other suggestions&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Many Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;</comment>
                            <comment id="3348653" author="dmitry.agranat" created="Thu, 20 Aug 2020 07:55:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;Since the requested logs are no longer available for the covering time we&apos;ve collected &lt;tt&gt;diagnostic.data&lt;/tt&gt;, we won&apos;t be able to grep for specific information you&apos;ve proposed in your last comment. However, I&apos;d like to step back to better understand the issue you are reporting.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The mongos seems to continuously keep on accumulating memory overtime until the mongos service is restarted. &lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Given what we saw via the &lt;tt&gt;diagnostic.data&lt;/tt&gt;, the resident memory grows by ~50 MB per day. With 8 GB RAM, it will take 163 days to use all memory. &lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Could you clarify why the mongoS service is being restarted given the trend of 163 days to get to 8 GB?&lt;/li&gt;
	&lt;li&gt;Where do you see that mongoS continuously keep on accumulating memory overtime?&lt;/li&gt;
	&lt;li&gt;What is the memory utilization today?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Just a reminder that memory is being managed by the OS and OS does not &quot;kick out&quot; those memory pages unless new data comes in *and *memory pressure is applied on the older pages. Since I do not see any memory pressure (we hardly use 20% of the total RAM size), I do not expect OS to release memory back to the system.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3347935" author="stephenpaul2727@gmail.com" created="Wed, 19 Aug 2020 20:16:11 +0000"  >&lt;p&gt;Hi Dmitry,&lt;/p&gt;

&lt;p&gt;Unfortunately, we only have 7 days worth of log retention available. So can&apos;t provide mongos logs during that time. But the issue also seems to be happening now. Also the logs have sensitive data because of queries.&#160; Is there a specific thread/task you are interested in? I can exclude COMMAND lines if that is okay?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Yes, the resident memory shouldn&apos;t be growing by mongos process. we wonder why mongos is trying to reserve that resident memory and not releasing it.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;</comment>
                            <comment id="3346918" author="dmitry.agranat" created="Wed, 19 Aug 2020 13:16:56 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;, we would also need the requested mongoS server logs covering the time when &lt;tt&gt;heapProfiling&lt;/tt&gt; was enabled. Please upload them to the same directory.&lt;/p&gt;

&lt;p&gt;Also, from a quick look at the &lt;tt&gt;diagnostic.data&lt;/tt&gt;, I see that the resident memory of mongod process grows by about 50 MB per day. From a couple of days we have data for, this is translated into 1450 MB at the start and 1550 MB after ~2 days. Do these numbers make sense to you and this is what the issue is?&lt;/p&gt;</comment>
                            <comment id="3341359" author="stephenpaul2727@gmail.com" created="Sat, 15 Aug 2020 23:32:45 +0000"  >&lt;p&gt;Hi Jon,&#160;&lt;/p&gt;

&lt;p&gt;I have uploaded the mongos diag tgz file.&#160;&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;</comment>
                            <comment id="3337015" author="jonathan.streets" created="Thu, 13 Aug 2020 14:42:39 +0000"  >&lt;p&gt;hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/66bd41ce-da5b-4531-b4a4-73c5070c69eb.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Jon&lt;/p&gt;</comment>
                            <comment id="3333745" author="stephenpaul2727@gmail.com" created="Tue, 11 Aug 2020 21:45:43 +0000"  >&lt;p&gt;Hi Jon, is there a mongo DB&apos;s private dump location link? (for ftdc data?) could you let me know?&lt;/p&gt;


&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Stephen&lt;/p&gt;</comment>
                            <comment id="3318777" author="stephenpaul2727@gmail.com" created="Mon, 3 Aug 2020 22:55:38 +0000"  >&lt;p&gt;Thanks jon for the info, I will attach the diagnostic data once we captured memory increase on mongos&lt;/p&gt;</comment>
                            <comment id="3315527" author="jonathan.streets" created="Fri, 31 Jul 2020 14:01:15 +0000"  >&lt;p&gt;hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=stephenpaul2727%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;stephenpaul2727@gmail.com&quot;&gt;stephenpaul2727@gmail.com&lt;/a&gt;,&lt;br/&gt;
 thank you for the information so far. In order to investigate this issue further we would need to obtain information from the mongos running with the &lt;tt&gt;--setParameter heapProfilingEnabled=true&lt;/tt&gt; option. Importantly, this does impact performance by up to 30% for CPU-bound workloads and is not recommended for production use. So, please consider carefully if you&apos;d prefer to go this route or not.&lt;/p&gt;

&lt;p&gt;After the problem reoccurs with heap profiling enabled, you can disable it and upload the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory, and the mongos server logs to this ticket. The data in that directory is known as the FTDC data, and is described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt; Regards,&lt;br/&gt;
 Jon&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1474732">SERVER-50944</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1413560">SERVER-49697</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="278293" name="RoutingTableHistory.png" size="343987" author="dmitry.agranat@mongodb.com" created="Mon, 14 Sep 2020 18:17:45 +0000"/>
                            <attachment id="278526" name="chunks-col-stats.txt" size="92716" author="stephenpaul2727@gmail.com" created="Tue, 15 Sep 2020 21:32:21 +0000"/>
                            <attachment id="278176" name="image-2020-09-13-23-41-29-108.png" size="6509" author="stephenpaul2727@gmail.com" created="Mon, 14 Sep 2020 06:41:29 +0000"/>
                            <attachment id="278177" name="image-2020-09-13-23-48-11-445.png" size="15328" author="stephenpaul2727@gmail.com" created="Mon, 14 Sep 2020 06:48:12 +0000"/>
                            <attachment id="274908" name="normalmongos.png" size="68240" author="stephenpaul2727@gmail.com" created="Fri, 21 Aug 2020 04:18:56 +0000"/>
                            <attachment id="274909" name="problematicmongos.png" size="70124" author="stephenpaul2727@gmail.com" created="Fri, 21 Aug 2020 04:18:56 +0000"/>
                            <attachment id="271948" name="serverstatus-ttp01.txt" size="17499" author="stephenpaul2727@gmail.com" created="Thu, 30 Jul 2020 00:25:37 +0000"/>
                            <attachment id="278295" name="stacks.html.zip" size="111390" author="dmitry.agranat@mongodb.com" created="Mon, 14 Sep 2020 18:20:36 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 31 Jul 2020 14:01:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 4 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 4 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>jonathan.streets@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>randolph@mongodb.com</customfieldvalue>
            <customfieldvalue>stephenpaul2727@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxxd0n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8rcf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxwz9z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>