<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:12:35 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-6749] What exactly causes &quot;can&apos;t find index in storeCurrentLocs&quot;</title>
                <link>https://jira.mongodb.org/browse/SERVER-6749</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Every now and then we run into errors like these :&lt;/p&gt;

&lt;p&gt;Fri Aug 10 11:29:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;Balancer&amp;#93;&lt;/span&gt; moveChunk result: &lt;/p&gt;
{ errmsg: &quot;can&apos;t find index in storeCurrentLocs&quot;, ok: 0.0 }
&lt;p&gt;Fri Aug 10 11:29:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;Balancer&amp;#93;&lt;/span&gt; balancer move failed: &lt;/p&gt;
{ errmsg: &quot;can&apos;t find index in storeCurrentLocs&quot;, ok: 0.0 }

&lt;p&gt;On top of that my config.mongos collection is slowly filling up with documents so those are not cleaned up (should they be?). For example I see 71 documents there while there are 20 mongos connected currently.&lt;/p&gt;

&lt;p&gt;Any idea what can cause these sorts of issues? Can killing mongos processes cause this for example? &lt;/p&gt;</description>
                <environment></environment>
        <key id="46786">SERVER-6749</key>
            <summary>What exactly causes &quot;can&apos;t find index in storeCurrentLocs&quot;</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="spencer@mongodb.com">Spencer Brody</assignee>
                                    <reporter username="remonvv">Remon van Vliet</reporter>
                        <labels>
                    </labels>
                <created>Fri, 10 Aug 2012 09:55:09 +0000</created>
                <updated>Wed, 10 Dec 2014 23:19:21 +0000</updated>
                            <resolved>Tue, 20 Aug 2013 15:26:39 +0000</resolved>
                                    <version>2.0.7</version>
                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="665365" author="justanyone" created="Mon, 21 Jul 2014 16:00:10 +0000"  >&lt;p&gt;Just had this happen.  Noticed that chunk migration had stopped for some number of days.  Last week, for various reasons, had to kill all monogd processes, some of which it urned out were rebuilding indexes at the time.  this led to a huge hassle, had to repairDatabase on those shards / replicasets.&lt;/p&gt;

&lt;p&gt;Also, noticed that some indexs were missing after this repair.  Had to apply so them agin with ensureIndex().&lt;/p&gt;

&lt;p&gt;Flash forward to this week, kept seeing no balancing and these silly messages likedthe above &quot;can&apos;t find index in storeCurrentLocs.&quot;&lt;/p&gt;

&lt;p&gt;SOLVED:   reIndex didn&apos;t work, but  I noticed that in th eoutput it listed indexes on each shard.  Some of the shards were missing an index, wihch turneed out to be the shard key index.  Have now issued a  command, ensureIndex() for that shardkey index, and it has been rebuilt.  Now not seeing error anymore.  &lt;/p&gt;

&lt;p&gt;Note that I had to run dbRepirDatabase after this because had another error:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2014-07-21T10:58:20.941-0500 [conn43697] myDBnameHere:myCollectionHere  0xf812a6 0xf2f092 0xf18f47 0xf190cc 0x7b9a0b 0xb86276 0xb8cd17 0x99efb0 0x9a1769 0x9a22da 0x9a2a75 0x9a5d80 0x9b4174 0x9b5753 0x9b645e 0xc4da55 0xac964e 0xace300 0x7bae2f 0xf3f429 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xf812a6]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x1a2) [0xf2f092]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x137) [0xf18f47]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod() [0xf190cc]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x2cb) [0x7b9a0b]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x1e86) [0xb86276]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo14UpdateExecutor7executeEv+0x67) [0xb8cd17]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod() [0x99efb0]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo18WriteBatchExecutor10execUpdateERKNS_12BatchItemRefEPNS_7BSONObjEPPNS_16WriteErrorDetailE+0x69) [0x9a1769]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0x24a) [0x9a22da]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x615) [0x9a2a75]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo8WriteCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x410) [0x9a5d80]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9b4174]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xad3) [0x9b5753]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2be) [0x9b645e]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xa55) [0xc4da55]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod() [0xac964e]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x4b0) [0xace300]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xef) [0x7bae2f]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x5a9) [0xf3f429]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;TO DO WORK for MONGO:  When this error occurs, additional text in the error message, something like, &quot;This could be caused by missing indexes on some shards.&quot; would be helpful.&lt;/p&gt;</comment>
                            <comment id="381171" author="remonvv" created="Mon, 15 Jul 2013 11:44:32 +0000"  >&lt;p&gt;Today we had the same error which led to a catastrophic failure for one of our customers. Version 2.0.8. I will open a new issue once I have collected all logs and such but I wanted to be able to refer to this one and vice versa. Basically scenario is as follows in this case :&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Every other day we scale a cluster up from 1 to many shards (10 in this case) at noon, and back down to 1 shard in the early morning.&lt;/li&gt;
	&lt;li&gt;For the upscale we invoke X (9 in this case) addShard commands concurrently and allow the balancer to redistribute chunks&lt;/li&gt;
	&lt;li&gt;For the downscale we invoke X (9 in tnis case) removeShards sequentially (so, one at a time to avoid another apparent mongos/config state issue)&lt;/li&gt;
	&lt;li&gt;Version is 2.0.8&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I&apos;m aware this is an atypical scenario given this frequency of capacity changes but we&apos;re not doing anything beyond invoking admin commands. We suspect either the cluster is left in a broken state after downscale or the upscale itself provokes the errors (&quot;can&apos;t find index in storeCurrentLocs&quot;). Either way it is not able to move chunks to other shards but it did enough to produce inconsistencies in the data somehow (our user account count went from ~617k to ~890k without any users being added indicating some sort of duplication).&lt;/p&gt;

&lt;p&gt;In any case, I&apos;ll follow up with an issue in Community Private.&lt;/p&gt;</comment>
                            <comment id="153621" author="remonvv" created="Wed, 15 Aug 2012 15:54:34 +0000"  >&lt;p&gt;Hi Spencer. Unfortunately the logs are gone for that period. It&apos;s on a test environment that doesn&apos;t back up the logs. I&apos;ll make sure we store all the mongos logs when it occurs again. For now I suppose you can close this issue since I cannot provide further information. I&apos;ll report an actual issue next time with more information. Thanks for the support &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="153616" author="spencer" created="Wed, 15 Aug 2012 15:36:54 +0000"  >&lt;p&gt;Hi Remon - any update on getting that log data so we can continue investigating what went wrong?  I&apos;d like logs from the mongos that was seeing the problem, as well as from nodes that were the primaries at the time for each shard&apos;s replica set.  As stated before, I&apos;d like to get the time period covering the full time that you were seeing the problem, as well as several hours before it started and after it was resolved.&lt;/p&gt;</comment>
                            <comment id="152656" author="remonvv" created="Mon, 13 Aug 2012 08:25:54 +0000"  >&lt;p&gt;I might be able to find the log data. It happens rather randomly. I understand we ideally shouldn&apos;t edit metadata but we need a fast solution and DNS tends to not be that. This seemed simplest but we&apos;ll look into alternatives. Still this seems like a state issue that shouldn&apos;t be allowed to happen.&lt;/p&gt;</comment>
                            <comment id="152422" author="spencer" created="Fri, 10 Aug 2012 20:07:43 +0000"  >&lt;p&gt;Do you know when exactly this started happening?  Do you have logs from that time period?  We&apos;d like to have several hours worth of logs from before and after the problem appeared to help our debugging.&lt;/p&gt;


&lt;p&gt;We strongly discourage editing config data directly.  Instead of using IP addresses in your configuration directly, it would be better to use elastic hostnames/DNS hostnames that can be updated when the IP addresses of the shards change, without having to modify the config data.&lt;/p&gt;</comment>
                            <comment id="152213" author="remonvv" created="Fri, 10 Aug 2012 10:26:49 +0000"  >&lt;p&gt;Updating the config.shards collection so that the chunks are assigned to the correct (or wrong, i guess) shard kickstarts the balancing again. So basically the most important question is how do we end up in this state in the first place.&lt;/p&gt;</comment>
                            <comment id="152212" author="remonvv" created="Fri, 10 Aug 2012 10:21:45 +0000"  >&lt;p&gt;And more info, it seems that the data is on the wrong shard. If we connect to shard0006 directly we find 26 million users there but the chunks collection has all those chunks on shard0000. The only somewhat sketchy thing we do is that we overwrite the IP addresses of the config.shards collection with the new IPs if the shard machines (we have a rather unique usecase where we bring up large clusters for a TV show and bring them down again, when they go up EC2 assigns new IP addresses hence this &quot;hack&quot;. It could be solved with naming). This is done when all mongos processes are down. The IP addresses are correct when this error occurs.&lt;/p&gt;</comment>
                            <comment id="152210" author="remonvv" created="Fri, 10 Aug 2012 10:08:51 +0000"  >&lt;p&gt;Related aborts in config.changelog :&lt;/p&gt;


&lt;p&gt;{ &quot;_id&quot; : &quot;xxxxxx.eu-west-1.compute.internal-2012-08-10T09:10:34-1&quot;, &quot;server&quot; : &quot;xxxxxx.eu-west-1.compute.internal&quot;, &quot;clientAddr&quot; : &quot;x.x.x.x:60276&quot;, &quot;time&quot; : ISODate(&quot;2012-08-10T09:10:34.741Z&quot;), &quot;what&quot; : &quot;moveChunk.from&quot;, &quot;ns&quot; : &quot;user.user.users&quot;, &quot;details&quot; : { &quot;min&quot; : &lt;/p&gt;
{ &quot;_h&quot; : 417566254 }
&lt;p&gt;, &quot;max&quot; : &lt;/p&gt;
{ &quot;_h&quot; : 447392415 }
&lt;p&gt;, &quot;step1&quot; : 800, &quot;step2&quot; : 604, &quot;note&quot; : &quot;aborted&quot; } }&lt;br/&gt;
{ &quot;_id&quot; : &quot;xxxxxx.eu-west-1.compute.internal-2012-08-10T09:10:37-3&quot;, &quot;server&quot; : &quot;xxxxxx.eu-west-1.compute.internal&quot;, &quot;clientAddr&quot; : &quot;x.x.x.x:46468&quot;, &quot;time&quot; : ISODate(&quot;2012-08-10T09:10:37.067Z&quot;), &quot;what&quot; : &quot;moveChunk.from&quot;, &quot;ns&quot; : &quot;user.user.users&quot;, &quot;details&quot; : { &quot;min&quot; : &lt;/p&gt;
{ &quot;_h&quot; : 417566254 }
&lt;p&gt;, &quot;max&quot; : &lt;/p&gt;
{ &quot;_h&quot; : 447392415 }
&lt;p&gt;, &quot;step1&quot; : 0, &quot;step2&quot; : 341, &quot;note&quot; : &quot;aborted&quot; } }&lt;/p&gt;</comment>
                            <comment id="152208" author="remonvv" created="Fri, 10 Aug 2012 09:58:31 +0000"  >&lt;p&gt;To add to this, I think the issue is related to a stale lock (either cause or consequence) :&lt;/p&gt;

&lt;p&gt;The collection that is no longer balancing is found in the config.locks collection and the same migrate keeps looping between state 0 and state 2 but as shown in the error never succeeds :&lt;/p&gt;

&lt;p&gt;{ &quot;_id&quot; : &quot;user.user.users&quot;, &quot;process&quot; : &quot;xxxxxx.eu-west-1.compute.internal:27018:1344589833:30611607&quot;, &quot;state&quot; : 2, &quot;ts&quot; : ObjectId(&quot;5024dab6618939edbeef7e0e&quot;), &quot;when&quot; : ISODate(&quot;2012-08-10T09:56:06.492Z&quot;), &quot;who&quot; : &quot;xxxxxx.eu-west-1.compute.internal:27018:1344589833:30611607:conn36:820470076&quot;, &quot;why&quot; : &quot;migrate-&lt;/p&gt;
{ _h: 417566254 }
&lt;p&gt;&quot; }&lt;br/&gt;
mongos&amp;gt; db.locks.find({_id:&quot;user.user.users&quot;})&lt;br/&gt;
{ &quot;_id&quot; : &quot;user.user.users&quot;, &quot;process&quot; : &quot;xxxxxx.eu-west-1.compute.internal:27018:1344589833:30611607&quot;, &quot;state&quot; : 0, &quot;ts&quot; : ObjectId(&quot;5024dab6618939edbeef7e0e&quot;), &quot;when&quot; : ISODate(&quot;2012-08-10T09:56:06.492Z&quot;), &quot;who&quot; : &quot;xxxxxx.eu-west-1.compute.internal:27018:1344589833:30611607:conn36:820470076&quot;, &quot;why&quot; : &quot;migrate-&lt;/p&gt;
{ _h: 417566254 }
&lt;p&gt;&quot; }&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 10 Aug 2012 20:07:43 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 30 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 30 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>justanyone</customfieldvalue>
            <customfieldvalue>remonvv</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrns13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrgn67:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10709</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsyhkv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>