<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:10:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-25922] Replication fails due to to many open files or out of memmory</title>
                <link>https://jira.mongodb.org/browse/SERVER-25922</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have 2 running Servers in replication.&lt;/p&gt;

&lt;p&gt;We are adding a 3rd server. The data seems to get synced without any problems, but the index fails to get build with either &quot;to many open files&quot; (although the limit here is set to 512000, recommended are 64000) or because of out of memmory.&lt;/p&gt;

&lt;p&gt;Attached you can find the log-outputs of both.&lt;/p&gt;

&lt;p&gt;I tested with version 3.2.4 and 3.2.9 on the affected instance.&lt;br/&gt;
The cluster is overall version 3.2.4 and running without problems.&lt;/p&gt;
</description>
                <environment>see attached additional-info</environment>
        <key id="313359">SERVER-25922</key>
            <summary>Replication fails due to to many open files or out of memmory</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="KralMar">Kral Markus [X]</reporter>
                        <labels>
                    </labels>
                <created>Thu, 1 Sep 2016 10:08:20 +0000</created>
                <updated>Tue, 4 Oct 2016 15:48:29 +0000</updated>
                            <resolved>Tue, 4 Oct 2016 15:48:29 +0000</resolved>
                                    <version>3.2.4</version>
                    <version>3.2.9</version>
                                                    <component>Replication</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1399810" author="thomas.schubert" created="Tue, 4 Oct 2016 15:48:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=KralMar&quot; class=&quot;user-hover&quot; rel=&quot;KralMar&quot;&gt;KralMar&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Since we haven&apos;t heard back from you, I assume that the ulimit settings explained the issue. Regarding the OOM kills, you have been affected by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-20306&quot; title=&quot;75% excess memory usage under WiredTiger during stress test&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-20306&quot;&gt;&lt;del&gt;SERVER-20306&lt;/del&gt;&lt;/a&gt;, which was fixed in MongoDB 3.2.10. If this is still an issue for you after upgrading, please let us know and we will continue to investigate.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1381850" author="thomas.schubert" created="Mon, 12 Sep 2016 12:56:23 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=KralMar&quot; class=&quot;user-hover&quot; rel=&quot;KralMar&quot;&gt;KralMar&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The index build is failing to open /srv/mongodb/data/_tmp/extsort.976 and I see WiredTiger has 45 currently open files, which brings the total number suspiciously close to the default ulimit setting of 1024.&lt;/p&gt;

&lt;p&gt;Would you please double check that the user running mongod has the correct ulimit settings?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;cat /proc/&amp;lt;mongod-pid&amp;gt;/limits&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1381741" author="kralmar" created="Mon, 12 Sep 2016 09:40:23 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;what is the current status here?&lt;/p&gt;

&lt;p&gt;Thanks&lt;br/&gt;
Markus&lt;/p&gt;</comment>
                            <comment id="1375395" author="kralmar" created="Fri, 2 Sep 2016 07:09:47 +0000"  >&lt;p&gt;Hi Ram&#243;n,&lt;/p&gt;

&lt;p&gt;thanks for your quick response.&lt;br/&gt;
I uploaded the requested files.&lt;/p&gt;

&lt;p&gt;As the environment is not that critical and we do have a running replica-set with 2 members and a backup, &lt;br/&gt;
I would wait for your instructions before trying anything further on my own.&lt;/p&gt;

</comment>
                            <comment id="1374476" author="ramon.fernandez" created="Thu, 1 Sep 2016 14:15:46 +0000"  >&lt;p&gt;I understand your concern &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=KralMar&quot; class=&quot;user-hover&quot; rel=&quot;KralMar&quot;&gt;KralMar&lt;/a&gt;. You can see the data that it collects &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/ftdc/ftdc_mongod.cpp#L314&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;, which is essentially the data produced by the following &lt;tt&gt;mongo&lt;/tt&gt; shell commands:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.serverStatus({tcmalloc: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;rs.status()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.getSiblingDB(&apos;local&apos;).oplog.rs.stats()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({getCmdLineOpts: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({buildInfo: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({hostInfo: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This data is gathered at periodic intervals by the server, compressed, and stored inside the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory. It contains no collection data.&lt;/p&gt;</comment>
                            <comment id="1374398" author="kralmar" created="Thu, 1 Sep 2016 13:01:38 +0000"  >&lt;p&gt;Hi Ram&#243;n,&lt;/p&gt;

&lt;p&gt;we are more than happy to provide you with everything you need,&lt;br/&gt;
however, wee need to know what kind of sensitive data the diagnostic.data includes because our database does includes personal data that cannot be easily shared (We could get into law-problems).&lt;/p&gt;

&lt;p&gt;Kind Regards&lt;br/&gt;
Markus&lt;/p&gt;</comment>
                            <comment id="1374379" author="ramon.fernandez" created="Thu, 1 Sep 2016 12:26:14 +0000"  >&lt;p&gt;Sorry to hear you&apos;re having trouble adding a third node &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=KralMar&quot; class=&quot;user-hover&quot; rel=&quot;KralMar&quot;&gt;KralMar&lt;/a&gt;. I don&apos;t see a &quot;smoking gun&quot; in the information you already sent, so I&apos;d like to ask you for the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The output of &lt;tt&gt;free&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;The contents of the &lt;tt&gt;/srv/mongodb/data/diagnostic.data&lt;/tt&gt; directory&lt;/li&gt;
	&lt;li&gt;The full log file for the initial sync attempts (&lt;tt&gt;/srv/mongodb/log/mongodb.log&lt;/tt&gt;)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/2d0c146f-18db-4e7e-a7f3-7e8de433d3a6.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; so you can share this information privately with us.&lt;/p&gt;

&lt;p&gt;If this issue is critical for you, I would recommend you consider one or both of these workarounds:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Looks like this is a virtual machine, so temporarily allocate more memory to it&lt;/li&gt;
	&lt;li&gt;Lower the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/configuration-options/#storage.wiredTiger.engineConfig.cacheSizeGB&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;size of the WiredTiger cache&lt;/a&gt; to, say 4GB&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I can&apos;t guarantee that these workarounds will address the issue since the data may indicate there&apos;s a bug somewhere, but trying won&apos;t hurt either. If you decide to try, please make sure you keep the existing contents of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory somewhere (uploading them to us is sufficient) &amp;#8211; if the initial sync succeeds with the workaround we&apos;ll need to compare this data before and after to understand what the issue is.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="137275" name="additional_info" size="1103" author="KralMar" created="Thu, 1 Sep 2016 10:08:20 +0000"/>
                            <attachment id="137276" name="mongodb_config" size="389" author="KralMar" created="Thu, 1 Sep 2016 10:08:20 +0000"/>
                            <attachment id="137277" name="oom_messages" size="17498" author="KralMar" created="Thu, 1 Sep 2016 10:08:20 +0000"/>
                            <attachment id="137278" name="startup_after_failure" size="3082" author="KralMar" created="Thu, 1 Sep 2016 10:08:20 +0000"/>
                            <attachment id="137279" name="too_many_open_files" size="6342" author="KralMar" created="Thu, 1 Sep 2016 10:08:20 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 1 Sep 2016 12:26:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 19 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 19 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>KralMar</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjx7r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hspbgf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsel8f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>