<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:45:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-17735] index build failed, ! _progressMeter.isActive()</title>
                <link>https://jira.mongodb.org/browse/SERVER-17735</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Most of this leadup shouldn&apos;t matter, but I&apos;m adding it for completeness.&lt;/p&gt;

&lt;p&gt;We were running mongo 2.4.10 until recently.&lt;br/&gt;
We have 3 mongo DB machines and 2 arbiters running in a replication set.&lt;br/&gt;
I upgraded each machine to 2.6.8.&lt;br/&gt;
When I went to upgrade the 3&apos;rd Db (after all other machines were upgraded) it ran into some problems, likely related to ulimits etc.&lt;/p&gt;

&lt;p&gt;So, I removed it from the replica set. &lt;br/&gt;
I upgraded the auth data as per: &lt;a href=&quot;http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/&lt;/a&gt;&lt;br/&gt;
I then got a new fresh machine, installed 2.6.8 on it, and tried to resync it.&lt;br/&gt;
We ran in to some problems with various limits (ulimits, and total system memory mapping limit), but eventually resolved those.&lt;/p&gt;

&lt;p&gt;Now, when the replica tries to resync, the log has errors all over, similar to those at the bottom of the post and it never finishes syncing. It might make it up to 26% of the 2T disk, but then drops to 6% again. &lt;/p&gt;

&lt;p&gt;2015-03-22T06:57:19.526+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;FileAllocator&amp;#93;&lt;/span&gt; allocating new datafile /db/mongodb/worplay-for-waterlink_meteor_com.0, filling with zeroes...&lt;br/&gt;
2015-03-22T06:57:19.529+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;FileAllocator&amp;#93;&lt;/span&gt; done allocating datafile /db/mongodb/worplay-for-waterlink_meteor_com.0, size: 16MB,  took 0.002 secs&lt;br/&gt;
2015-03-22T06:57:19.535+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; build index on: worplay-for-waterlink_meteor_com.system.users properties: { v: 1, key: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, name: &quot;&lt;em&gt;id&lt;/em&gt;&quot;, ns: &quot;worplay-for-waterlink_meteor_com.system.users&quot; }&lt;br/&gt;
2015-03-22T06:57:19.535+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt;    building index using bulk method&lt;br/&gt;
2015-03-22T06:57:19.535+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; worplay-for-waterlink_meteor_com Assertion failure ! _progressMeter.isActive() src/mongo/db/curop.cpp 154&lt;br/&gt;
2015-03-22T06:57:19.542+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; worplay-for-waterlink_meteor_com 0x1205431 0x11a7229 0x118b53e 0xa1a460 0x8eea05 0x8f13c2 0x8e70f2 0x8e8d47 0x910a62 0xea0d28 0xea1dad 0xea3177 0xec3844 0xec38f0 0xec3c0a 0x1249dc9 0x7f3e79b1a182 0x7f3e78e1efbd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x1205431&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x11a7229&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x118b53e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo5CurOp10setMessageEPKcSsyi+0x1c0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa1a460&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb+0x1a5) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8eea05&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb+0x8e2) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8f13c2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE+0x922) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8e70f2&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12IndexCatalog17ensureHaveIdIndexEv+0x427) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x8e8d47&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo6Cloner2goERNS_6Client7ContextERKSsRKNS_12CloneOptionsEPSt3setISsSt4lessISsESaISsEERSsPi+0x1ad2) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x910a62&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl17_initialSyncCloneERNS_6ClonerERKSsRKSt4listISsSaISsEEb+0x318) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xea0d28&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl12_initialSyncEv+0x7ad) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xea1dad&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x37) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xea3177&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0x124) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xec3844&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x30) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xec38f0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0xaa) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xec3c0a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x1249dc9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f3e79b1a182&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f3e78e1efbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
2015-03-22T06:57:19.542+0000 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; index build failed. spec: { v: 1, key: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, name: &quot;&lt;em&gt;id&lt;/em&gt;&quot;, ns: &quot;worplay-for-waterlink_meteor_com.system.users&quot; } error: 0 assertion src/mongo/db/curop.cpp:154&lt;/p&gt;

&lt;p&gt;Thanks... happy to add more detail if it&apos;s needed.&lt;/p&gt;</description>
                <environment></environment>
        <key id="191921">SERVER-17735</key>
            <summary>index build failed, ! _progressMeter.isActive()</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="sam.kleinman">Sam Kleinman</assignee>
                                    <reporter username="mbrewer">Matthew Brewer</reporter>
                        <labels>
                    </labels>
                <created>Wed, 25 Mar 2015 17:44:10 +0000</created>
                <updated>Wed, 8 Apr 2015 17:46:36 +0000</updated>
                            <resolved>Wed, 8 Apr 2015 16:14:02 +0000</resolved>
                                    <version>2.6.8</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="876019" author="ramon.fernandez" created="Wed, 8 Apr 2015 17:46:24 +0000"  >&lt;p&gt;Thanks for the update &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mbrewer&quot; class=&quot;user-hover&quot; rel=&quot;mbrewer&quot;&gt;mbrewer&lt;/a&gt;. Going forward you may want to consider fixing the long keys, as it&apos;s preferable to run with &lt;tt&gt;failIndexKeyTooLong&lt;/tt&gt; set to true to make sure queries that use indices return complete results.&lt;/p&gt;</comment>
                            <comment id="875888" author="mbrewer" created="Wed, 8 Apr 2015 16:04:18 +0000"  >&lt;p&gt;We&apos;ve resolved this. Indeed it is the IDs...&lt;/p&gt;

&lt;p&gt;The way to confirm post 2.6 migration is to turn off ID length checking via the failIndexKeyToLong flag.&lt;br/&gt;
The command is: &quot;db.getSiblingDB(&apos;admin&apos;).runCommand( &lt;/p&gt;
{ setParameter: 1, failIndexKeyTooLong: false }
&lt;p&gt; )&quot;&lt;br/&gt;
We ran this on every machine before a resync, and no longer got this failure (we&apos;re still having other issues, but unrelated)&lt;/p&gt;

&lt;p&gt;Thanks for the help, and sorry for the bug-report for a non-bug.&lt;/p&gt;
</comment>
                            <comment id="865341" author="mbrewer" created="Thu, 26 Mar 2015 19:18:16 +0000"  >&lt;p&gt;From my bug:&lt;br/&gt;
&quot;I upgraded the auth data as per: &lt;a href=&quot;http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/&lt;/a&gt;&quot;&lt;br/&gt;
So... yup, got that one.&lt;/p&gt;

&lt;p&gt;Sorry. I&apos;m asking If there is a way to check whether the id issue is the cause &lt;b&gt;retroactively&lt;/b&gt;... meaning after we no longer have a 2.4.x instance in the cluster. We now have all 2.6.8, not 2.4 anymore. It would&apos;ve been nice if I&apos;d captured logs when I ran that command originally, but I did not, and I don&apos;t trust human memory enough to say what the real result was. Given that, I was hoping for a way to know if this &quot;id&quot; issue is actually the problem at this point.&lt;/p&gt;

&lt;p&gt;If I hadn&apos;t upgraded auth, I could add 2.4.* machine to the replica-set, just to run this test, but I have upgraded it so that won&apos;t work.&lt;/p&gt;

&lt;p&gt;Is there some other way to make progress on root causing this?&lt;/p&gt;</comment>
                            <comment id="865174" author="samk" created="Thu, 26 Mar 2015 17:37:31 +0000"  >&lt;p&gt;You can run &lt;tt&gt;db.upgradeCheckAllDBs()&lt;/tt&gt; at any point on a copy of the data managed by 2.4.x to see if there are problems in the data that are still unresolved. &lt;/p&gt;

&lt;p&gt;Additionally, I&apos;m sorry I missed this earlier but have you run the auth schema upgrade process on this data? See the documentation &lt;a href=&quot;http://docs.mongodb.org/manual/release-notes/2.6-upgrade-authorization/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; for more information on this process.&lt;/p&gt;</comment>
                            <comment id="864823" author="mbrewer" created="Thu, 26 Mar 2015 13:36:46 +0000"  >&lt;p&gt;Thanks for the help/response!&lt;/p&gt;

&lt;p&gt;We host users, so the answer to #1 is &quot;I don&apos;t know&quot;. It&apos;s pretty likely that some of our users do generate their own ids though.&lt;/p&gt;

&lt;p&gt;I did run db.upgradeCheckAllDBs()... but honestly I don&apos;t remember exactly what it said. I vaguely recall that it did not return a &quot;no&quot; or a &quot;yes&quot; answer and instead did something nonsensical... but I could easily be wrong. I had previously run it on a test DB on which it had the same behavior for that call, and did work.&lt;br/&gt;
Given the slew of known bugs in 2.4.10 I&apos;m used to tripping over I didn&apos;t bother recording or reporting that behavior, sad to say but we&apos;re pretty used to mongo not behaving very well. Sorry.&lt;/p&gt;

&lt;p&gt;Is there some way to check whether this is the issue retroactively?&lt;/p&gt;</comment>
                            <comment id="864051" author="samk" created="Wed, 25 Mar 2015 18:50:12 +0000"  >&lt;p&gt;Thanks for this report. I have a couple of additional questions: &lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Are the values stored in your collection&apos;s &lt;tt&gt;_id&lt;/tt&gt; fields generated by your application, or do you use the default &lt;tt&gt;ObjectId&lt;/tt&gt; values. If you use custom values, what values do you use and how do you generate them?&lt;/li&gt;
	&lt;li&gt;Did you run &lt;tt&gt;db.upgradeCheckAllDBs()&lt;/tt&gt; &lt;a href=&quot;http://docs.mongodb.org/master/reference/method/db.upgradeCheckAllDBs/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;docs&lt;/a&gt; before upgrading? What was the outcome of this operation?&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 25 Mar 2015 18:50:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 45 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 45 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>mbrewer</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>sam.kleinman</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl9xr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs7gxb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;I&apos;m not sure that you can repro, as this probably relates to specific DB layout.&lt;/p&gt;

&lt;p&gt;Here&apos;s the setup we&apos;re running on though:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Mongo 2.6.8&lt;/li&gt;
	&lt;li&gt;ubuntu 14.04 LTS&lt;/li&gt;
	&lt;li&gt;uname -a -&amp;gt; Linux production-db-b1v0 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux&lt;/li&gt;
	&lt;li&gt;Running on AWS, m3.large, with 2 1T SSD ebs volumes running raid 0&lt;/li&gt;
&lt;/ul&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsg60n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>