<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:42:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16802] Order of balancer chunk moves depends on order of config.collections</title>
                <link>https://jira.mongodb.org/browse/SERVER-16802</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h6&gt;&lt;a name=&quot;Background&quot;&gt;&lt;/a&gt;Background&lt;/h6&gt;

&lt;p&gt;Currently, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9080534036974a91067cef0ace3aff60c65a123b/src/mongo/s/balance.cpp#L372-L502&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Balancer::_doBalanceRound()&lt;/a&gt; and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/9080534036974a91067cef0ace3aff60c65a123b/src/mongo/s/balancer_policy.cpp#L348&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;BalancerPolicy::balance()&lt;/a&gt; together find candidate chunks with the following pseudocode:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;For each collection in config.collections:
	&lt;ul&gt;
		&lt;li&gt;Look for chunks on draining shards&lt;/li&gt;
		&lt;li&gt;Look for tag violating chunks&lt;/li&gt;
		&lt;li&gt;Look for imbalance within each tag&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;



&lt;h6&gt;&lt;a name=&quot;Problem&quot;&gt;&lt;/a&gt;Problem&lt;/h6&gt;

&lt;p&gt;This approach means that the balancer can attempt &quot;regular&quot; imbalance chunk moves &lt;b&gt;before&lt;/b&gt; tag violation chunk moves, which can be &lt;b&gt;before&lt;/b&gt; shard drain moves.  This is counter-intuitive, because users expect:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;chunks to commence moving off draining shards very soon after removeShard has been run, and&lt;/li&gt;
	&lt;li&gt;tag violation chunks to be moved ahead of regular imbalance moves.&lt;/li&gt;
&lt;/ul&gt;



&lt;h6&gt;&lt;a name=&quot;Impact&quot;&gt;&lt;/a&gt;Impact&lt;/h6&gt;

&lt;p&gt;This is worsened by:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;A lot of imbalanced sharded collections, but a shard being drained has chunks only in collections near the &lt;em&gt;end&lt;/em&gt; of the list (e.g. caused by tags).  In this case, the user will observe many &quot;irrelevant&quot; non-draining chunk moves prior to (and in between) draining chunk moves (the pattern repeats each balancing round).&lt;/li&gt;
	&lt;li&gt;Where earlier lower-priority moves can&apos;t complete because the TO shard can&apos;t satisfy the w:majority pre-commit check.  Then each problematic earlier chunk move fails after 10 hours, causing higher-priority moves to happen a lot later.  In the case of a shard drain, this can make it take a very long time, and the system to outwardly appear nearly idle (very little balance/drain/move activity).  Again, this is counter-intuitive because the user believes that they have just instructed the system to start draining (but it &quot;isn&apos;t&quot;).&lt;/li&gt;
&lt;/ol&gt;



&lt;h6&gt;&lt;a name=&quot;Suggestion&quot;&gt;&lt;/a&gt;Suggestion&lt;/h6&gt;

&lt;p&gt;Perhaps the code could be rearranged along the lines of:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;For each collection in config.collections:
	&lt;ul&gt;
		&lt;li&gt;Look for chunks on draining shards&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;For each &lt;em&gt;remaining&lt;/em&gt; collection:
	&lt;ul&gt;
		&lt;li&gt;Look for tag violating chunks&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;For each &lt;em&gt;remaining&lt;/em&gt; collection:
	&lt;ul&gt;
		&lt;li&gt;Look for imbalance within each tag&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Each collection would still have at most 1 chunk move per balance round, which should allow lower-priority moves to make progress (e.g. the balancer won&apos;t be &quot;hogged&quot; when draining a large shard, or when adding new tags).  But it would also have the benefit of ensuring that &amp;#8212; irrespective of the order of config.collections &amp;#8212;  draining moves are given priority over tag violation moves, which are given priority over imbalance moves.&lt;/p&gt;</description>
                <environment></environment>
        <key id="178118">SERVER-16802</key>
            <summary>Order of balancer chunk moves depends on order of config.collections</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kevin.pulo@mongodb.com">Kevin Pulo</assignee>
                                    <reporter username="kevin.pulo@mongodb.com">Kevin Pulo</reporter>
                        <labels>
                            <label>balancer</label>
                    </labels>
                <created>Mon, 12 Jan 2015 02:13:56 +0000</created>
                <updated>Wed, 30 May 2018 13:29:29 +0000</updated>
                            <resolved>Fri, 16 Mar 2018 13:36:43 +0000</resolved>
                                    <version>2.8.0-rc4</version>
                                    <fixVersion>3.4.15</fixVersion>
                    <fixVersion>3.6.4</fixVersion>
                    <fixVersion>3.7.4</fixVersion>
                                    <component>Sharding</component>
                                        <votes>1</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="1844617" author="xgen-internal-githook" created="Mon, 26 Mar 2018 01:13:19 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;kevin.pulo@mongodb.com&apos;, &apos;name&apos;: &apos;Kevin Pulo&apos;, &apos;username&apos;: &apos;devkev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16802&quot; title=&quot;Order of balancer chunk moves depends on order of config.collections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16802&quot;&gt;&lt;del&gt;SERVER-16802&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28981&quot; title=&quot;Sharding balancer prefers shards in a specific order when moving chunks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28981&quot;&gt;&lt;del&gt;SERVER-28981&lt;/del&gt;&lt;/a&gt; Balancer consider shards and collections in random order&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 651b3e017ce880d9ddbebb400af621c61d8c7389)&lt;br/&gt;
Branch: v3.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/af4ea84a50cf35473c77ea2b27f19c63afd43bc1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/af4ea84a50cf35473c77ea2b27f19c63afd43bc1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1838017" author="xgen-internal-githook" created="Mon, 19 Mar 2018 10:28:07 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;kevin.pulo@mongodb.com&apos;, &apos;name&apos;: &apos;Kevin Pulo&apos;, &apos;username&apos;: &apos;devkev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16802&quot; title=&quot;Order of balancer chunk moves depends on order of config.collections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16802&quot;&gt;&lt;del&gt;SERVER-16802&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28981&quot; title=&quot;Sharding balancer prefers shards in a specific order when moving chunks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28981&quot;&gt;&lt;del&gt;SERVER-28981&lt;/del&gt;&lt;/a&gt; Balancer consider shards and collections in random order&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 9850f1f190f13fb5bfd229e35d55d8fee3adc58f)&lt;br/&gt;
Branch: v3.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/651b3e017ce880d9ddbebb400af621c61d8c7389&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/651b3e017ce880d9ddbebb400af621c61d8c7389&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1836278" author="xgen-internal-githook" created="Fri, 16 Mar 2018 13:34:05 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;kevin.pulo@mongodb.com&apos;, &apos;name&apos;: &apos;Kevin Pulo&apos;, &apos;username&apos;: &apos;devkev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16802&quot; title=&quot;Order of balancer chunk moves depends on order of config.collections&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16802&quot;&gt;&lt;del&gt;SERVER-16802&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28981&quot; title=&quot;Sharding balancer prefers shards in a specific order when moving chunks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28981&quot;&gt;&lt;del&gt;SERVER-28981&lt;/del&gt;&lt;/a&gt; Balancer consider shards and collections in random order&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9850f1f190f13fb5bfd229e35d55d8fee3adc58f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9850f1f190f13fb5bfd229e35d55d8fee3adc58f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1265560" author="kaloian.manassiev" created="Mon, 16 May 2016 21:17:51 +0000"  >&lt;p&gt;With the changes to move the balancer to the CSRS primary we split the balancer loop into a &quot;policy&quot; part, which returns list of all chunks, which need to be moved on that round. Currently this list is still sorted based on the collection namespace, but it would be fairly easy to make it sorted additionally first based on the draining status, then based on the tag violation status and finally on the collection name.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    <customfieldvalue key="14340"><![CDATA[v3.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000atZPzIAM]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 12 Jan 2015 05:16:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 46 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-629</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 46 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>kevin.pulo@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlf5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8u0n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>156282</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2110">Sharding 2018-02-26</customfieldvalue>
    <customfieldvalue id="2166">Sharding 2018-03-12</customfieldvalue>
    <customfieldvalue id="2167">Sharding 2018-03-26</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11861" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>User Summary</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11856"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsgcb3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>