<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 02:54:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-423] db.&lt;dname&gt;.ensureIndex( { uid: 1 }, {unique: true} ) eats up all ram and swap</title>
                <link>https://jira.mongodb.org/browse/SERVER-423</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When creating a NON-Unique index, index is created normally.  On a collection with 3.5M documents:&lt;br/&gt;
I did a  db.&amp;lt;dname&amp;gt;.ensureIndex( &lt;/p&gt;
{ uid: 1 }
&lt;p&gt; ):&lt;/p&gt;


&lt;p&gt;Fri Nov 13 13:44:24 cpu: 4000   0       0%&lt;br/&gt;
Fri Nov 13 13:44:25 building new index on &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt; for &amp;lt;dbname&amp;gt;.Farm...&lt;br/&gt;
Fri Nov 13 13:44:25 Buildindex &amp;lt;dbname&amp;gt;,Farm idxNo:1 { ns: &quot;&amp;lt;dbname&amp;gt;.Farm&quot;, key: &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt;, name: &quot;uid_1&quot; }&lt;br/&gt;
Fri Nov 13 13:48:22 done for 3517687 records 237.078secs&lt;br/&gt;
Fri Nov 13 13:44:25 insert &amp;lt;dnbame&amp;gt;.system.indexes 237259ms&lt;br/&gt;
Fri Nov 13 13:48:22 cpu: 238225 237265  99%&lt;/p&gt;



&lt;p&gt;I did a db.&amp;lt;dbname&amp;gt;.dropIndexes();&lt;br/&gt;
{&lt;br/&gt;
        &quot;nIndexesWas&quot; : 2,&lt;br/&gt;
        &quot;msg&quot; : &quot;all indexes deleted for collection&quot;,&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;


&lt;p&gt;Then  db.&amp;lt;dname&amp;gt;.ensureIndex( &lt;/p&gt;
{ uid: 1 }
&lt;p&gt;, &lt;/p&gt;
{unique: true}
&lt;p&gt; )&lt;/p&gt;


&lt;p&gt;Fri Nov 13 13:51:10 cpu: 4000   0       0%&lt;br/&gt;
Fri Nov 13 13:51:10 building new index on &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt; for &amp;lt;dname&amp;gt;.Farm...&lt;br/&gt;
Fri Nov 13 13:51:11 Buildindex &amp;lt;dbname&amp;gt;.Farm idxNo:1 { ns: &quot;&amp;lt;dname&amp;gt;.Farm&quot;, key: &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt;, name: &quot;uid_1&quot;, unique: true }&lt;br/&gt;
Fri Nov 13 14:05:28 User Exception E11000 duplicate key errorindex: &amp;lt;dname&amp;gt;.Farm.$uid_1  dup key: { : 501272713 }&lt;br/&gt;
Fri Nov 13 14:05:38  &amp;lt;dbname&amp;gt;.system.indexes  Caught Assertion insert, continuing&lt;br/&gt;
Fri Nov 13 13:51:10 insert &amp;lt;dbname&amp;gt;.system.indexes exception userassert:E11000 duplicate key errorindex: &amp;lt;dbname&amp;gt;.Farm.$uid_1  dup key: { : 501272713 } 867252ms&lt;br/&gt;
Fri Nov 13 14:05:38 cpu: 867851 867252  99%&lt;/p&gt;


&lt;p&gt;During this second run, mongo went almost 2GB into swap and as you can see, took almost 4 times as long before even detecting a conflict. I had to restart mongodb to clear up the ram usage.. it was using all system ram (RES 7.2G).   The whole collection size is around 16G.&lt;/p&gt;

&lt;p&gt;&amp;gt; db.&amp;lt;dbname&amp;gt;.dropIndexes();&lt;br/&gt;
{&lt;br/&gt;
        &quot;nIndexesWas&quot; : 1,&lt;br/&gt;
        &quot;msg&quot; : &quot;all indexes deleted for collection&quot;,&lt;br/&gt;
        &quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;Now, even adding a regular index causes massive swap use:&lt;br/&gt;
 db.&amp;lt;dbname&amp;gt;.ensureIndex( &lt;/p&gt;
{ uid: 1 }
&lt;p&gt; )&lt;/p&gt;

&lt;p&gt;top - 14:16:50 up 20:28,  4 users,  load average: 1.50, 0.89, 0.93&lt;br/&gt;
Tasks:  66 total,   2 running,  64 sleeping,   0 stopped,   0 zombie&lt;br/&gt;
Cpu(s): 42.7%us,  0.3%sy,  0.0%ni, 49.3%id,  0.7%wa,  0.0%hi,  0.0%si,  6.9%st&lt;br/&gt;
Mem:   7872040k total,  7855940k used,    16100k free,      104k buffers&lt;br/&gt;
Swap:  8388600k total,  1092068k used,  7296532k free,  5337924k cached&lt;/p&gt;

&lt;p&gt;  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND&lt;br/&gt;
 5480 mongo     19   0 22.1g 7.3g 5.1g S   82 96.6   0:13.91 mongod&lt;/p&gt;


&lt;p&gt;It completes, but ram is not freed:&lt;/p&gt;

&lt;p&gt;Fri Nov 13 14:13:34 cpu: 4000   0       0%&lt;br/&gt;
Fri Nov 13 14:13:34 building new index on &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt; for &amp;lt;dbname&amp;gt;.Farm...&lt;br/&gt;
Fri Nov 13 14:13:34 Buildindex &amp;lt;dbname&amp;gt;.Farm idxNo:1 { ns: &quot;&amp;lt;dbname&amp;gt;.Farm&quot;, key: &lt;/p&gt;
{ uid: 1.0 }
&lt;p&gt;, name: &quot;uid_1&quot; }&lt;br/&gt;
Fri Nov 13 14:16:58 done for 3517687 records 203.592secs&lt;br/&gt;
Fri Nov 13 14:13:34 insert &amp;lt;dbname&amp;gt;.system.indexes 203648ms&lt;br/&gt;
Fri Nov 13 14:16:58 cpu: 204073 203655  99%&lt;/p&gt;

</description>
                <environment>AWS Linux m1.large instance.  x64 &lt;br/&gt;
Linux ip-10-244-51-31 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux&lt;br/&gt;
Thu Nov 12 17:50:50 Mongo DB : starting : pid = 2413 port = 27017 dbpath = /mnt/mdb master = 1 slave = 0  64-bit&lt;br/&gt;
Thu Nov 12 17:50:50 db version v1.1.3, pdfile version 4.5&lt;br/&gt;
Thu Nov 12 17:50:50 git version: aa8b12e13fbbdfb293d94786dcc0df589cabef99&lt;br/&gt;
Thu Nov 12 17:50:50 sys info: Linux ofc-n1.10gen.com 2.6.23.17-88.fc7 #1 SMP Thu May 15 00:02:29 EDT 2008 x86_64&lt;br/&gt;
&lt;br/&gt;
</environment>
        <key id="10876">SERVER-423</key>
            <summary>db.&lt;dname&gt;.ensureIndex( { uid: 1 }, {unique: true} ) eats up all ram and swap</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="eliot">Eliot Horowitz</assignee>
                                    <reporter username="azilber">Alexey Zilber</reporter>
                        <labels>
                    </labels>
                <created>Fri, 13 Nov 2009 15:38:36 +0000</created>
                <updated>Tue, 12 Jul 2016 00:28:17 +0000</updated>
                            <resolved>Tue, 6 Apr 2010 13:30:29 +0000</resolved>
                                    <version>1.1.3</version>
                                                    <component>Index Maintenance</component>
                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>0</watches>
                                                                                                                <comments>
                            <comment id="13470" author="eliot" created="Tue, 6 Apr 2010 13:30:29 +0000"  >&lt;p&gt;this should be fixed in 1.4&lt;/p&gt;</comment>
                            <comment id="12375" author="eliot" created="Wed, 17 Feb 2010 22:56:46 +0000"  >&lt;p&gt;some changes in 1.3.2&lt;br/&gt;
anyway you could try that?&lt;/p&gt;</comment>
                            <comment id="11166" author="azilber" created="Tue, 17 Nov 2009 16:26:37 +0000"  >&lt;p&gt;I can give you access to the instance where I have the csv file, you&apos;re welcome to see for yourself.&lt;/p&gt;

&lt;p&gt;-Alex&lt;/p&gt;</comment>
                            <comment id="11149" author="auto" created="Fri, 13 Nov 2009 21:08:31 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Eliot Horowitz&apos;, &apos;email&apos;: &apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: BDONObjBuilder::apendAsNumber&lt;br/&gt;
better fix for &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-423&quot; title=&quot;db.&amp;lt;dname&amp;gt;.ensureIndex( { uid: 1 }, {unique: true} ) eats up all ram and swap&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-423&quot;&gt;&lt;del&gt;SERVER-423&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://github.com/mongodb/mongo/commit/ca785a040086939005a88b06fa77f9133751450e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://github.com/mongodb/mongo/commit/ca785a040086939005a88b06fa77f9133751450e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="11148" author="eliot" created="Fri, 13 Nov 2009 20:54:15 +0000"  >&lt;p&gt;I cannot reproduce this.  &lt;br/&gt;
I&apos;m seeing a consistent 10% slowdown for unique, which is about what you should expect.&lt;br/&gt;
Can you make a reproducible test that we can try?&lt;/p&gt;</comment>
                            <comment id="11143" author="auto" created="Fri, 13 Nov 2009 17:26:18 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Eliot Horowitz&apos;, &apos;email&apos;: &apos;eliot@10gen.com&apos;}
&lt;p&gt;Message: some index building code cleaning looking at &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-423&quot; title=&quot;db.&amp;lt;dname&amp;gt;.ensureIndex( { uid: 1 }, {unique: true} ) eats up all ram and swap&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-423&quot;&gt;&lt;del&gt;SERVER-423&lt;/del&gt;&lt;/a&gt;, but can&apos;t see anything yet&lt;br/&gt;
&lt;a href=&quot;http://github.com/mongodb/mongo/commit/7db9f7ee87339b703d309f97fa20c17e243c5bad&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://github.com/mongodb/mongo/commit/7db9f7ee87339b703d309f97fa20c17e243c5bad&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 13 Nov 2009 17:26:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        13 years, 46 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            13 years, 46 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>azilber</customfieldvalue>
            <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrpumv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrim3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22231</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0jun:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>