<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:22:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30060] Make the balancer gather storage statistics only for shards which have `maxSize` set</title>
                <link>https://jira.mongodb.org/browse/SERVER-30060</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The sharding balancer currently issues &lt;tt&gt;listDatabases&lt;/tt&gt; against every single shard in order to access the &lt;tt&gt;totalSize&lt;/tt&gt; value. This value is used for ensuring that a shard&apos;s storage &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/manage-sharded-cluster-balancer/#change-the-maximum-storage-size-for-a-given-shard&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;maxSize&lt;/tt&gt;&lt;/a&gt; is not exceeded for customers which have that value set.&lt;/p&gt;

&lt;p&gt;The &lt;tt&gt;listDatabases&lt;/tt&gt; call is quite heavy, especially for nodes with large number of databases/collections since it will &lt;tt&gt;fstat&lt;/tt&gt; every single file under the instance.&lt;/p&gt;

&lt;p&gt;There are a number of optimizations we can make in order to make this statistics gathering less expensive (listed in order of preference):&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;b&gt;Only gather storage statistics for shards which have &lt;tt&gt;maxSize&lt;/tt&gt; set (implemented by this ticket)&lt;/b&gt;&lt;/li&gt;
	&lt;li&gt;Issue the &lt;tt&gt;listDatabases&lt;/tt&gt; call in parallel against all shards so it doesn&apos;t take so different shards&apos; execution overlaps&lt;/li&gt;
	&lt;li&gt;Cache the per-shard statistics so that they are not collected on every single round/moveChunk invocation&lt;/li&gt;
	&lt;li&gt;Collect the per-shard statistics asynchronously so that multiple concurrent moveChunk requests can benefit&lt;/li&gt;
	&lt;li&gt;Add a parameter to &lt;tt&gt;listDatabases&lt;/tt&gt; to allow it to return cached data size instead of every time {{fstat}}ing all the files&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="401707">SERVER-30060</key>
            <summary>Make the balancer gather storage statistics only for shards which have `maxSize` set</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</assignee>
                                    <reporter username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</reporter>
                        <labels>
                    </labels>
                <created>Fri, 7 Jul 2017 20:34:10 +0000</created>
                <updated>Mon, 30 Oct 2023 23:15:22 +0000</updated>
                            <resolved>Mon, 17 Jul 2017 17:38:29 +0000</resolved>
                                    <version>3.4.6</version>
                    <version>3.5.9</version>
                                    <fixVersion>3.4.7</fixVersion>
                    <fixVersion>3.5.11</fixVersion>
                                    <component>Sharding</component>
                                        <votes>3</votes>
                                    <watches>17</watches>
                                                                                                                <comments>
                            <comment id="1630765" author="kaloian.manassiev" created="Tue, 25 Jul 2017 01:48:07 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=arie%40netskope.com&quot; class=&quot;user-hover&quot; rel=&quot;arie@netskope.com&quot;&gt;arie@netskope.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The optimization which we implemented is the first one in the list from the description - to not collect storage utilization statistics for shards which do not have the &lt;tt&gt;maxSize&lt;/tt&gt; setting. We chose to go with it because it is the least risky to backport and also because we estimated that it would cover the majority of our customers&apos; use cases, due to heterogeneous hardware being more of an exception.&lt;/p&gt;

&lt;p&gt;Hope this helps and apologies for the confusion.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1630740" author="arie@netskope.com" created="Tue, 25 Jul 2017 00:51:03 +0000"  >&lt;p&gt;After looking at the code change, it was not clear to me which one of the optimizations was chosen. Can you please share?&lt;/p&gt;</comment>
                            <comment id="1624100" author="xgen-internal-githook" created="Mon, 17 Jul 2017 20:37:12 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;kaloianm&apos;, u&apos;name&apos;: u&apos;Kaloian Manassiev&apos;, u&apos;email&apos;: u&apos;kaloian.manassiev@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30060&quot; title=&quot;Make the balancer gather storage statistics only for shards which have `maxSize` set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30060&quot;&gt;&lt;del&gt;SERVER-30060&lt;/del&gt;&lt;/a&gt; Do not gather shard disk usage statistics unless &apos;maxSize&apos; is set&lt;/p&gt;

&lt;p&gt;(cherry picked from commit e0136739285c097a7da59ba54d6bcd109bb184b5)&lt;br/&gt;
Branch: v3.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/060473c2794e1d86dc4e4988c031b9c7ebda8297&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/060473c2794e1d86dc4e4988c031b9c7ebda8297&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1623812" author="xgen-internal-githook" created="Mon, 17 Jul 2017 17:12:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;kaloianm&apos;, u&apos;name&apos;: u&apos;Kaloian Manassiev&apos;, u&apos;email&apos;: u&apos;kaloian.manassiev@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30060&quot; title=&quot;Make the balancer gather storage statistics only for shards which have `maxSize` set&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30060&quot;&gt;&lt;del&gt;SERVER-30060&lt;/del&gt;&lt;/a&gt; Do not gather shard disk usage statistics unless &apos;maxSize&apos; is set&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e0136739285c097a7da59ba54d6bcd109bb184b5&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/e0136739285c097a7da59ba54d6bcd109bb184b5&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="538472">SERVER-34819</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="14340"><![CDATA[v3.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000XNLhlIAH, 500A000000aQcAuIAK]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 17 Jul 2017 17:12:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 29 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 29 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>arie@netskope.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htaiof:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht2f9z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1727">Sharding 2017-07-31</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11856"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hta4r3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>