<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:08:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-5312] Resync fails due to out of memory</title>
                <link>https://jira.mongodb.org/browse/SERVER-5312</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We newly initiated a replica set, but the to-be secondary never gets out of &quot;RECOVERING&quot; state, as the mongod process is killed by oom-killer in the middle of resync (seemingly last step of resync- when building secondary indexes, to be precise) and start from scratch every time.&lt;/p&gt;

&lt;p&gt;journal is turned on, vm.overcommit_memory is set to 1, as suggested before.&lt;/p&gt;

&lt;p&gt;Right now, testing &quot;echo -17 &amp;gt; /proc/`cat /var/run/mongodb.pid`/oom_adj&quot; (and &quot;swapoff -a&quot;), but every trial takes hours.&lt;/p&gt;

&lt;p&gt;The data size is 10x larger than the physical memory, it seems unlikely that simply doubling the RAM would fix the problem, as the heuristics of oom-killer is rather unpredictable.&lt;/p&gt;

&lt;p&gt;I&apos;d like to know what triggers this failure, and what I should keep in mind.&lt;/p&gt;

&lt;p&gt;What should we do to get resync done?&lt;/p&gt;</description>
                <environment>Linux 3.0.18</environment>
        <key id="32976">SERVER-5312</key>
            <summary>Resync fails due to out of memory</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="siddharth.singh@10gen.com">siddharth.singh@10gen.com</assignee>
                                    <reporter username="kenn">Kenn Ejima</reporter>
                        <labels>
                            <label>replicaset</label>
                    </labels>
                <created>Wed, 14 Mar 2012 18:39:44 +0000</created>
                <updated>Wed, 15 Aug 2012 14:04:17 +0000</updated>
                            <resolved>Wed, 11 Jul 2012 19:39:50 +0000</resolved>
                                    <version>2.0.3</version>
                                                    <component>Replication</component>
                                        <votes>1</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="142033" author="dan@10gen.com" created="Wed, 11 Jul 2012 19:39:50 +0000"  >&lt;p&gt;This was found and fixed by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6414&quot; title=&quot;use regular file io, not mmap for external sort&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6414&quot;&gt;&lt;del&gt;SERVER-6414&lt;/del&gt;&lt;/a&gt;.  Going into 2.0.7 and 2.2.0.&lt;/p&gt;</comment>
                            <comment id="124356" author="siddharth.singh@10gen.com" created="Wed, 30 May 2012 15:27:21 +0000"  >&lt;p&gt;I had turned swapoff to try repro it faster as well. Didn&apos;t see the repro though. It took a while for the secondaries to sycn but it happened successfully.&lt;/p&gt;</comment>
                            <comment id="124105" author="kenn" created="Tue, 29 May 2012 22:26:52 +0000"  >&lt;p&gt;&quot;swapoff -a&quot; is to quickly let success or fail those experiments, that&apos;s all. I didn&apos;t want to let it run days or weeks by thrashing.&lt;/p&gt;

&lt;p&gt;what is your swap size? in our case, swap size was 256MB (default of Linode).&lt;/p&gt;</comment>
                            <comment id="124103" author="siddharth.singh@10gen.com" created="Tue, 29 May 2012 22:18:45 +0000"  >&lt;p&gt;No repro yet. Tried it with index size of 3.5, 9 and 21 GB on a 1 GB machine and didn&apos;t see anything. I will go ahead and close this one for now as &quot;Unable to Reproduce&quot;. Please feel free to reopen if you come across a reproducible test case. Thanks for reporting this.&lt;/p&gt;
</comment>
                            <comment id="114464" author="siddharth.singh@10gen.com" created="Fri, 27 Apr 2012 20:48:02 +0000"  >&lt;p&gt;Unable to reproduce so far. Tried with a ~3.5GB index size on a VM with 1 GB machine. Will update as soon as I get a repro case.&lt;/p&gt;</comment>
                            <comment id="105779" author="kenn" created="Mon, 2 Apr 2012 18:20:48 +0000"  >&lt;p&gt;Here&apos;s the stats on the new 2.0 instances, it&apos;s a lot more compact than the old 1.8 for the same data. We discarded the old instances (upgrading mongodb was the purpose of the replication / migration) so the information on them is not available.&lt;/p&gt;

&lt;p&gt;&amp;gt; db.stats()&lt;br/&gt;
{&lt;br/&gt;
	&quot;db&quot; : &quot;my_production&quot;,&lt;br/&gt;
	&quot;collections&quot; : 15,&lt;br/&gt;
	&quot;objects&quot; : 41682353,&lt;br/&gt;
	&quot;avgObjSize&quot; : 290.2968635191972,&lt;br/&gt;
	&quot;dataSize&quot; : 12100256340,&lt;br/&gt;
	&quot;storageSize&quot; : 14245875712,&lt;br/&gt;
	&quot;numExtents&quot; : 123,&lt;br/&gt;
	&quot;indexes&quot; : 34,&lt;br/&gt;
	&quot;indexSize&quot; : 7570043936,&lt;br/&gt;
	&quot;fileSize&quot; : 25691160576,&lt;br/&gt;
	&quot;nsSizeMB&quot; : 16,&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;&amp;gt; db.files.stats()&lt;br/&gt;
{&lt;br/&gt;
	&quot;ns&quot; : &quot;my_production.uploads&quot;,&lt;br/&gt;
	&quot;count&quot; : 16494054,&lt;br/&gt;
	&quot;size&quot; : 4854031388,&lt;br/&gt;
	&quot;avgObjSize&quot; : 294.28977181716516,&lt;br/&gt;
	&quot;storageSize&quot; : 5727051776,&lt;br/&gt;
	&quot;numExtents&quot; : 32,&lt;br/&gt;
	&quot;nindexes&quot; : 4,&lt;br/&gt;
	&quot;lastExtentSize&quot; : 961069056,&lt;br/&gt;
	&quot;paddingFactor&quot; : 1.6499999999847526,&lt;br/&gt;
	&quot;flags&quot; : 1,&lt;br/&gt;
	&quot;totalIndexSize&quot; : 2346225840,&lt;br/&gt;
	&quot;indexSizes&quot; : &lt;/p&gt;
{
		&quot;_id_&quot; : 505930880,
		&quot;id2_1&quot; : 435870736,
		&quot;game_id_1_filename_1&quot; : 1108739184,
		&quot;expires_at_1&quot; : 295685040
	}
&lt;p&gt;,&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;db.system.index.find() returned nothing.&lt;/p&gt;

&lt;p&gt;There are too many collections and stats - I only list the collection that caused the problem. The &quot;game_id_1_filename_1&quot; index is like 1GB now but it was much larger before.&lt;/p&gt;

&lt;p&gt;Try reproducing on a server with small RAM capacity, it should give you a repro quicker, I think.&lt;/p&gt;

&lt;p&gt;Hope this helps.&lt;/p&gt;</comment>
                            <comment id="105284" author="siddharth.singh@10gen.com" created="Fri, 30 Mar 2012 17:49:32 +0000"  >&lt;p&gt;I would like to reproduce this case. Can you give us information:&lt;/p&gt;

&lt;p&gt;i) About the database and collection level stats by running (db.stats(), db.collection.stats()). &lt;br/&gt;
ii) The kind of indices you have and information from database indices (db.system.index.find()) &lt;/p&gt;

&lt;p&gt;I noticed that it says &quot;external sort used: &quot;. Presumably memory mapping these external sort files might be causing the memory usage to go up. I&quot;ll try to reproduce this case and see what&apos;s going on.&lt;/p&gt;</comment>
                            <comment id="104635" author="kenn" created="Thu, 29 Mar 2012 04:24:19 +0000"  >&lt;p&gt;Unfortunately, no. Luckily for us, doubling the RAM fixed the problem this time, but I know the bottom cause is still there and might have the same problem as our data set grow.&lt;/p&gt;

&lt;p&gt;To me it seems that the problem occurs when you have a large INDEX, probably larger than RAM.&lt;/p&gt;</comment>
                            <comment id="104633" author="desmortes" created="Thu, 29 Mar 2012 04:06:10 +0000"  >&lt;p&gt;I have the same problem and it&apos;s a production instance. I am running a Amazon&apos;s m1.medium 64-bit machine.&lt;/p&gt;

&lt;p&gt;@Kenn were you able to fix this problem?? If yes, please let me know, Thanks.&lt;/p&gt;</comment>
                            <comment id="99853" author="kenn" created="Sat, 17 Mar 2012 07:53:29 +0000"  >&lt;p&gt;We use linode.com, a Xen based VPS with unknown type of RAID (most likely RAID10). Their disk performance is fairly fast. steal was about 0 with sysstat, so it&apos;s not that other users on the same hardware occupied the resources.&lt;/p&gt;</comment>
                            <comment id="99845" author="eliot" created="Sat, 17 Mar 2012 06:40:00 +0000"  >&lt;p&gt;What hardware is this on?&lt;br/&gt;
What kind of drives?&lt;/p&gt;</comment>
                            <comment id="99044" author="kenn" created="Thu, 15 Mar 2012 07:05:20 +0000"  >&lt;p&gt;Yes, disk was saturated as detailed more at &lt;a href=&quot;http://groups.google.com/group/mongodb-user/browse_thread/thread/f4e4089b8439ff9f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://groups.google.com/group/mongodb-user/browse_thread/thread/f4e4089b8439ff9f&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;At that time iowait was like 90%. Before to that point, it was more like RES=4.0g, SHR=3.8g, but when running building the secondary indexes, SHR went down to 0 pretty rapidly and crashed hard (Even console couldn&apos;t accept login shell and had to restart physically, however it was virtualized and emulated).&lt;/p&gt;</comment>
                            <comment id="99027" author="eliot" created="Thu, 15 Mar 2012 05:10:18 +0000"  >&lt;p&gt;Is disk saturated?&lt;/p&gt;</comment>
                            <comment id="98801" author="kenn" created="Wed, 14 Mar 2012 19:13:31 +0000"  >&lt;p&gt;Follow up:&lt;/p&gt;

&lt;p&gt;Okay this is third trial (oom adjusted and swapoff -a), and this time mongod is not get killed, but progress has gone incredibly slow. Right now it&apos;s rebuilding secondary indexes:&lt;/p&gt;

&lt;p&gt;external sort used : 10 files  in 54 secs&lt;br/&gt;
570700/9432323	6%&lt;br/&gt;
575500/9432323	6%&lt;br/&gt;
579800/9432323	6%&lt;br/&gt;
...&lt;/p&gt;

&lt;p&gt;and it&apos;s taking forever.&lt;/p&gt;

&lt;p&gt;Looking at top, mongod eats up all the memory and only 1-2MB is left for file cache. VIRT=70g, RES=4.0g, SHR=0.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="43962">SERVER-6414</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="41915">SERVER-6141</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 15 Mar 2012 05:10:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 32 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 32 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kenn</customfieldvalue>
            <customfieldvalue>desmortes</customfieldvalue>
            <customfieldvalue>siddharth.singh@10gen.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hro9kf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriphj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22782</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0a53:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>