<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:17:12 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-8348] balancer locked in state 0</title>
                <link>https://jira.mongodb.org/browse/SERVER-8348</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Balancer seems to have gotten stuck after acquiring a lock which then stopped all balancing until we logged into the mongos and stop/started the balancer.&lt;/p&gt;

&lt;p&gt;mongos log-&lt;br/&gt;
Fri Jan 18 00:45:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;Balancer&amp;#93;&lt;/span&gt; distributed lock &apos;balancer/ms-db-4:27017:1354112926:1804289383&apos; acquired, ts : 50f8fd7e8d5ca2b8deff6383&lt;/p&gt;

&lt;p&gt;use config;&lt;br/&gt;
mongos&amp;gt; db.locks.find( &lt;/p&gt;
{ _id : &quot;balancer&quot; }
&lt;p&gt; ).pretty()&lt;br/&gt;
{&lt;br/&gt;
        &quot;_id&quot; : &quot;balancer&quot;,&lt;br/&gt;
        &quot;process&quot; : &quot;ms-db-4:27017:1354112926:1804289383&quot;,&lt;br/&gt;
        &quot;state&quot; : 0,&lt;br/&gt;
        &quot;ts&quot; : ObjectId(&quot;50f8fd7e8d5ca2b8deff6383&quot;),&lt;br/&gt;
        &quot;when&quot; : ISODate(&quot;2013-01-18T07:45:02.249Z&quot;),&lt;br/&gt;
        &quot;who&quot; : &quot;ms-db-4:27017:1354112926:1804289383:Balancer:846930886&quot;,&lt;br/&gt;
        &quot;why&quot; : &quot;doing balance round&quot;&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;we didn&apos;t notice this until today that we had no balancing happening.  &lt;/p&gt;</description>
                <environment>ubuntu 12.04.01 LTS</environment>
        <key id="63342">SERVER-8348</key>
            <summary>balancer locked in state 0</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="andre.defrere@mongodb.com">Andre de Frere</assignee>
                                    <reporter username="joep">joe piscitella</reporter>
                        <labels>
                    </labels>
                <created>Fri, 25 Jan 2013 23:17:28 +0000</created>
                <updated>Mon, 1 Apr 2013 14:03:06 +0000</updated>
                            <resolved>Mon, 1 Apr 2013 14:03:06 +0000</resolved>
                                    <version>2.2.1</version>
                                                    <component>Sharding</component>
                                        <votes>3</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="275258" author="joep" created="Mon, 25 Feb 2013 15:55:03 +0000"  >&lt;p&gt;Confirmed that it was a code bug and not a mongo bug.  Please close the issue.&lt;/p&gt;</comment>
                            <comment id="274793" author="andre.defrere" created="Mon, 25 Feb 2013 00:32:16 +0000"  >&lt;p&gt;Hi Joe,&lt;/p&gt;

&lt;p&gt;I&apos;ll set this to Waiting on User Input until we hear back from you on how your testing went.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andr&#233;&lt;/p&gt;</comment>
                            <comment id="273265" author="joep" created="Fri, 22 Feb 2013 17:37:35 +0000"  >&lt;p&gt;Andre, thank you for the feedback.&lt;br/&gt;
changed backup code and testing&lt;/p&gt;</comment>
                            <comment id="269666" author="andre.defrere" created="Tue, 19 Feb 2013 02:40:55 +0000"  >&lt;p&gt;Hi Joe,&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;seeing a lot of those &quot;did not propagate properly. :: caused by :: 8017 update not consistent&quot; errors in mongos logs.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Are you consistently seeing these messages at the time you are stopping the balancer before your backup?&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;the balancer is getting stop/started properly during the backups as after running the stop I do a find to get the state and echo it to the log which is showing stopped state = ture&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Do you check both that the balancer is stopped (or more accurately, has had a stop requested) and that the balancer is no longer balancing?  This is the difference between &lt;tt&gt;sh.getBalancerState()&lt;/tt&gt; and &lt;tt&gt;sh.isBalancerRunning()&lt;/tt&gt;.  The balancer can be requested to stop by changing the collection (as you are doing), but may still be in the middle of a balance round and therefore will still report as running (or the state key in the locks collection will be &amp;gt;0 for &lt;/p&gt;
{ _id : &apos;balancer&apos; }
&lt;p&gt; ).  For this reason, the &lt;tt&gt;sh.stopBalancer()&lt;/tt&gt; helper will wait until the balancer has actually finished before returning.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andr&#233;&lt;/p&gt;</comment>
                            <comment id="269206" author="joep" created="Mon, 18 Feb 2013 13:15:15 +0000"  >&lt;p&gt;seeing a lot of those &quot;did not propagate properly. :: caused by :: 8017 update not consistent&quot; errors in mongos logs.&lt;/p&gt;

&lt;p&gt;after running the dbhash on all three config servers the only difference is the changelog in all three config servers.&lt;/p&gt;

&lt;p&gt;the balancer is getting stop/started properly during the backups as after running the stop I do a find to get the state and echo it to the log which is showing stopped state = ture&lt;/p&gt;</comment>
                            <comment id="269007" author="andre.defrere" created="Mon, 18 Feb 2013 05:51:46 +0000"  >&lt;p&gt;Hi Joe,&lt;/p&gt;

&lt;p&gt;The &quot;o&quot; log file you have attached indicates that the balancer lock is being prevented from being set to 1 because the update is not consistent:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Mon Feb 11 00:43:43 [Balancer] warning: distributed lock &apos;balancer/db-mdb-12.:27017:1355258231:1804289383 did not propagate properly. :: caused by :: 8017 update not consistent  ns: config.locks query: { _id: &quot;balancer&quot;, state: 0, ts: ObjectId(&apos;5118a12b57a3e60537d10d90&apos;) } update: { $set: { state: 1, who: &quot;db-mdb-12.:27017:1355258231:1804289383:Balancer:846930886&quot;, process: &quot;db-mdb-12.:27017:1355258231:1804289383&quot;, when: new Date(1360568623601), why: &quot;doing balance round&quot;, ts: ObjectId(&apos;5118a12ff72247b5d1e98121&apos;) } } gle1: { updatedExisting: true, n: 1, connectionId: 211471, waited: 84, err: null, ok: 1.0 } gle2: { updatedExisting: false, n: 0, connectionId: 210743, waited: 26, err: null, ok: 1.0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Does this message appear every night during your backup routine?  Do you check the locks collection after updating to ensure the balancer is off?  Can you run a dbHash on the config database from each of the three config servers and ensure that the dbs have the same hash?  The command to do this looks like the following:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;use config&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.runCommand({dbHash:1})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;if this happens reproducibly, would you be able to run the dbHash on all three config servers after reproducing the issue?&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andr&#233;&lt;/p&gt;</comment>
                            <comment id="263203" author="joep" created="Mon, 11 Feb 2013 16:08:54 +0000"  >&lt;p&gt;This log does show some promise as to what is happening.  Shows some errors leading up to when the balancer is turned off for backups.&lt;/p&gt;

&lt;p&gt;Mon Feb 11 00:45:01 2013 Checking for a balancer....&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Balancer Found!&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Writing balancer stopped state to file&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Pre-backup balancer stopped state is : 0&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Stopping balancer....&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Checking current balancer state...&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Balancer stopped status is : true&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Checking balancer active state...&lt;br/&gt;
 Mon Feb 11 00:45:02 2013 Balancer active state is : 0&lt;/p&gt;</comment>
                            <comment id="262452" author="joep" created="Sun, 10 Feb 2013 13:36:53 +0000"  >&lt;p&gt;This is happening every night after the balancer is turned off for backups , $coll-&amp;gt;update(&lt;/p&gt;
{&quot;_id&quot; =&amp;gt; &quot;balancer&quot; }
&lt;p&gt;, { &apos;$set&apos; =&amp;gt; &lt;/p&gt;
{ &quot;stopped&quot;=&amp;gt; &quot;true&quot; }
&lt;p&gt; }); .  After turning the balancer back on it never comes to life again.  Each morning I restart it via the sh.start/stopBalancer in any mongos and it starts working again.&lt;/p&gt;

&lt;p&gt;The problem is there is no error so I am not sure what to attach.  &lt;/p&gt;</comment>
                            <comment id="257322" author="jantje.daniel" created="Mon, 4 Feb 2013 00:58:15 +0000"  >&lt;p&gt;Please can you attach the log file from the mongos covering the period in which you had this error.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="23972" name="mlog" size="6112" author="joep" created="Sun, 10 Feb 2013 13:52:35 +0000"/>
                            <attachment id="23989" name="o" size="8000" author="joep" created="Mon, 11 Feb 2013 16:08:54 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Feb 2013 00:58:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 51 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 51 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andre.defrere@mongodb.com</customfieldvalue>
            <customfieldvalue>jantje.daniel</customfieldvalue>
            <customfieldvalue>joep</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrn9gv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrltxr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>40909</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;above is the information from the mongos log when the lock was acquired and the output of the mongos locks.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmto7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>