<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:20:01 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29149] SHARDING CatalogCacheLoader + Balancer filling up to 2gb of log in ~4h</title>
                <link>https://jira.mongodb.org/browse/SERVER-29149</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have deployed a new production set-up using the same configuration that we have for the other one, however for this one the mongod configuration server keeps spamming us with these message non stop:&lt;/p&gt;

&lt;p&gt;2017-05-11T17:12:18.303+0000 I SHARDING &lt;span class=&quot;error&quot;&gt;&amp;#91;Balancer&amp;#93;&lt;/span&gt; Refreshing chunks for collection (collection) based on version 1|0||590245312abfdf91c6d415fe&lt;/p&gt;

&lt;p&gt;2017-05-11T17:12:18.303+0000 I SHARDING &lt;span class=&quot;error&quot;&gt;&amp;#91;CatalogCacheLoader-25&amp;#93;&lt;/span&gt; Refresh for collection (Collection) took 0 ms and found version 1|0||590245312abfdf91c6d415fe&lt;/p&gt;

&lt;p&gt;This lead us to having to rotate and delete all log after a 1 hour period. it is started in quiet mode with verbosity set to 0. The chunk size have been modified to 128mb and it&apos;s been like that since we&apos;ve set-up the server and I can&apos;t figure out why it&apos;s happening only on this 3 machine clusters &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Thanks !&lt;br/&gt;
St&#233;phane&lt;/p&gt;
</description>
                <environment></environment>
        <key id="382780">SERVER-29149</key>
            <summary>SHARDING CatalogCacheLoader + Balancer filling up to 2gb of log in ~4h</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="smarquis">Stephane Marquis</reporter>
                        <labels>
                    </labels>
                <created>Thu, 11 May 2017 17:14:44 +0000</created>
                <updated>Thu, 24 Aug 2017 04:23:50 +0000</updated>
                            <resolved>Mon, 17 Jul 2017 13:42:27 +0000</resolved>
                                    <version>3.4.4</version>
                                                    <component>Sharding</component>
                                        <votes>3</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1623505" author="kaloian.manassiev" created="Mon, 17 Jul 2017 13:42:03 +0000"  >&lt;p&gt;Thanks for the confirmation, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=oleg%40evergage.com&quot; class=&quot;user-hover&quot; rel=&quot;oleg@evergage.com&quot;&gt;oleg@evergage.com&lt;/a&gt;. I am going to close this ticket as duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28418&quot; title=&quot;make the split command on mongod return a stale version error if the requested chunk bounds are not found&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28418&quot;&gt;&lt;del&gt;SERVER-28418&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=smarquis&quot; class=&quot;user-hover&quot; rel=&quot;smarquis&quot;&gt;smarquis&lt;/a&gt;, if this issue persists for you for any reason, please open a new ticket and include a snippet from the relevant logs so we can investigate further.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1621308" author="oleg@evergage.com" created="Thu, 13 Jul 2017 17:54:46 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt; I can confirm that after the upgrade, the log volume has decreased from 12MB/min to 0.10MB/min on the primary config server that was affected.&lt;/p&gt;

&lt;p&gt;You can consider this resolved.&lt;/p&gt;</comment>
                            <comment id="1616270" author="kaloian.manassiev" created="Fri, 7 Jul 2017 14:30:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=smarquis&quot; class=&quot;user-hover&quot; rel=&quot;smarquis&quot;&gt;smarquis&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=oleg%40evergage.com&quot; class=&quot;user-hover&quot; rel=&quot;oleg@evergage.com&quot;&gt;oleg@evergage.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;MongoDB 3.4.6 has been released. Would it be possible to upgrade to this version and let me know if you are still seeing abnormal log generation?&lt;/p&gt;

&lt;p&gt;In the meantime we are looking for ways to reduce the amount of refresh information logging without sacrificing our ability to diagnose problems post factum.&lt;/p&gt;

&lt;p&gt;Thank you very much for your help and patience.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1606655" author="oleg@evergage.com" created="Mon, 26 Jun 2017 16:27:58 +0000"  >&lt;p&gt;I think that &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28418&quot; title=&quot;make the split command on mongod return a stale version error if the requested chunk bounds are not found&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28418&quot;&gt;&lt;del&gt;SERVER-28418&lt;/del&gt;&lt;/a&gt; is the root cause of massively spammy balancer logs. Can&apos;t wait for 3.4.6..&lt;/p&gt;</comment>
                            <comment id="1593790" author="oleg@evergage.com" created="Sat, 10 Jun 2017 22:05:53 +0000"  >&lt;p&gt;If I stop the balancer, the heavy refreshing behavior stops and log spam goes away. I think this might be caused by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29423&quot; title=&quot;Sharding balancer schedules multiple migrations with the same conflicting source or destination&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29423&quot;&gt;&lt;del&gt;SERVER-29423&lt;/del&gt;&lt;/a&gt;, so it is possible that you might see this only for clusters that have many collections that need to be balanced.&lt;/p&gt;</comment>
                            <comment id="1570205" author="smarquis" created="Fri, 12 May 2017 21:00:15 +0000"  >&lt;p&gt;I&apos;ve checked the last 3 rounds of log and the message are always :&lt;/p&gt;

&lt;p&gt;2017-05-12T20:58:27.298+0000 I SHARDING &lt;span class=&quot;error&quot;&gt;&amp;#91;Balancer&amp;#93;&lt;/span&gt; Refreshing chunks for collection (Collection) based on version 3|1||5908c72e40f2ee4a9952df08&lt;br/&gt;
2017-05-12T20:58:27.299+0000 I SHARDING &lt;span class=&quot;error&quot;&gt;&amp;#91;CatalogCacheLoader-77&amp;#93;&lt;/span&gt; Refresh for collection (Collection) took 0 ms and found version 3|1||5908c72e40f2ee4a9952df08&lt;/p&gt;


&lt;p&gt;The version are varying by collection but that&apos;s it :-/ i&apos;m not seeing any error &lt;/p&gt;</comment>
                            <comment id="1570197" author="kaloian.manassiev" created="Fri, 12 May 2017 20:54:52 +0000"  >&lt;p&gt;I am wondering whether the extra logging on the 3.4.4 cluster could be due to some error. Is the message for different collections or always the same by any chance?&lt;/p&gt;
</comment>
                            <comment id="1570185" author="smarquis" created="Fri, 12 May 2017 20:33:13 +0000"  >&lt;p&gt;Weirdly enough, on our other cluster (3.4.2) there&apos;s no message being sent at the 10 seconds interval. I&apos;ve validated the config server configuration file and there&apos;s nothing related to log verbosity or systemLog.quiet in there, they are started like that:&lt;/p&gt;

&lt;p&gt;mongod   10988 26.3 53.6 40478940 35370896 ?   Sl   Apr16 9899:15 /usr/bin/mongod -f /etc/mongod.conf&lt;br/&gt;
mongod   11075  0.7  3.0 3068788 2024848 ?     Sl   Apr16 265:14 /usr/bin/mongod -f /etc/mongod_config.conf&lt;/p&gt;

&lt;p&gt;There&apos;s not the quiet switch either, could it be related to one cluster being on CentOS 6 and the other one being on CentOS 7 ?&lt;/p&gt;

&lt;p&gt;Thanks &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; ! &lt;/p&gt;</comment>
                            <comment id="1569995" author="kaloian.manassiev" created="Fri, 12 May 2017 18:50:29 +0000"  >&lt;p&gt;I believe you need to use &lt;a href=&quot;https://docs.mongodb.com/manual/reference/configuration-options/#systemLog.quiet&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;systemLog.quiet&lt;/tt&gt;&lt;/a&gt; in the config file not as a parameter on the command line.&lt;/p&gt;</comment>
                            <comment id="1569843" author="smarquis" created="Fri, 12 May 2017 16:55:28 +0000"  >&lt;p&gt;Hi Kaloian,&lt;/p&gt;

&lt;p&gt;Could there be a bug with the quiet switch ? All the nodes are started as follow:&lt;/p&gt;

&lt;p&gt;mongod    7111  8.8 52.6 36635156 34640692 ?   Sl   May08 540:05 /usr/bin/mongod --quiet -f /etc/mongod.conf run&lt;br/&gt;
mongod   13080  1.3  0.1 592004 94944 ?        Sl   May10  42:54 /usr/bin/mongos --quiet -f /etc/mongos.conf&lt;br/&gt;
mongod   13143  0.8  0.6 1260896 413132 ?      Sl   May10  28:47 /usr/bin/mongod --quiet -f /etc/mongod_config.conf run&lt;/p&gt;

&lt;p&gt;This cluster was a new one (no upgrade) while the other one have more collection but the log partition has ~60g of log so it wasn&apos;t an issue on it. I have to leave for a few but I&apos;ll check it when I come back to see if the same amount of log is getting outputted.&lt;/p&gt;</comment>
                            <comment id="1569835" author="kaloian.manassiev" created="Fri, 12 May 2017 16:47:55 +0000"  >&lt;p&gt;Hi Stephane,&lt;/p&gt;

&lt;p&gt;Sorry that you are experiencing this problem. We will use this ticket to figure out a more efficient way to log refreshes on the config server and post an update.&lt;/p&gt;

&lt;p&gt;For now, unfortunately there is no way to disable these messages without &lt;a href=&quot;https://docs.mongodb.com/v3.2/reference/parameters/#param.quiet&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;disabling&lt;/a&gt; all the logging on the node. However, I would strongly recommend against this because it severely limits the ability to diagnose problems.&lt;/p&gt;

&lt;p&gt;In the mean time, can you please tell me whether this is a brand new cluster setup or you upgraded an existing cluster to 3.4.4? The reason I am asking is that I checked the code and the same logging is present in 3.4.2 as well. Is it that your older cluster just has a smaller number of sharded collections?&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1569683" author="smarquis" created="Fri, 12 May 2017 14:31:54 +0000"  >&lt;p&gt;Hello Kaloian, &lt;/p&gt;

&lt;p&gt;You are right our old cluster is 3.4.2 which is why we don&apos;t get them. &lt;/p&gt;

&lt;p&gt;There is no error shown on this particular cluster we have a total of :&lt;/p&gt;

&lt;p&gt;434 collections, unfortunately in almost all collection name there is sensitive information and going throught the log to change them all would take a lot of time. After further validation, once we clear all the log it last for a day (clean-up at 6 am and we get the warning that the log partition is full around 5h am) which would be ~1.7gb of log during that period (4mb * 434 collections ~1736mb) on a 2gb partition.&lt;/p&gt;

&lt;p&gt;Is there any way we can turn them off ? &lt;/p&gt;

&lt;p&gt;Thanks ! &lt;/p&gt;</comment>
                            <comment id="1569664" author="kaloian.manassiev" created="Fri, 12 May 2017 14:23:15 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=smarquis&quot; class=&quot;user-hover&quot; rel=&quot;smarquis&quot;&gt;smarquis&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The messages you have listed were introduced in version 3.4.4 as a result of making the metadata refresh happen asynchronously and block other operations. As a result we seem to have doubled the amount of log lines around each of the balancer&apos;s rounds (the first message was always there).&lt;/p&gt;

&lt;p&gt;I suspect that your old cluster is running 3.4.2 and the new cluster is running 3.4.4 that&apos;s why you are seeing it only on one of the clusters.&lt;/p&gt;

&lt;p&gt;The balancer round messages should show up once every 10 seconds and would be 2 per collection, so the logging impact per collection would be about 4MB per day. This is not great, but not anywhere close to 4GB.&lt;/p&gt;

&lt;p&gt;Are you seeing them show up more often and what is the number of collections that you have? Are there any errors shown? Would it be possible to attach a snippet from one of these logs?&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="366748">SERVER-28418</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 12 May 2017 14:23:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 30 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 30 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>oleg@evergage.com</customfieldvalue>
            <customfieldvalue>smarquis</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht7apr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hszgxr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Would really like to know :| &lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs1k93:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>