<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:45:16 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-17687] Fatal assertion after pthread_create fails with Resource temporarily unavailable</title>
                <link>https://jira.mongodb.org/browse/SERVER-17687</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I&apos;m trying to backup my mongodb collection to amazon s3 using spark. So my cluster only read data from collection in may threads. After few hours mongodb server crashes. See log in attachment. If you need more details, please ask, I will provide additional information.&lt;/p&gt;</description>
                <environment></environment>
        <key id="191271">SERVER-17687</key>
            <summary>Fatal assertion after pthread_create fails with Resource temporarily unavailable</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ramon.fernandez@mongodb.com">Ramon Fernandez Marina</assignee>
                                    <reporter username="dselivanov">Dmitriy Selivanov</reporter>
                        <labels>
                    </labels>
                <created>Mon, 23 Mar 2015 11:49:30 +0000</created>
                <updated>Wed, 24 Mar 2021 20:37:27 +0000</updated>
                            <resolved>Wed, 1 Apr 2015 20:27:05 +0000</resolved>
                                    <version>3.0.1</version>
                                                    <component>Stability</component>
                                        <votes>1</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="3682486" author="ramon.fernandez" created="Wed, 24 Mar 2021 20:37:27 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vasanth3g%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;vasanth3g@gmail.com&quot;&gt;vasanth3g@gmail.com&lt;/a&gt;, please note it&apos;s always better to open a new ticket than commenting on an old, closed one.&lt;/p&gt;

&lt;p&gt;That said, with the information above my initial assessment is that &lt;tt&gt;testmanager&lt;/tt&gt; is opening a lot of connections, but those are not being closed properly thus leaving the socket in CLOSE_WAIT, and at some point exhausting all available file descriptors. At that point, &lt;tt&gt;mongod&lt;/tt&gt; is unable to create new threads, and that&apos;s why you see the error message you&apos;re seeing.&lt;/p&gt;

&lt;p&gt;If you believe this could be a bug in &lt;tt&gt;mongod&lt;/tt&gt; feel free to open a new ticket, but &lt;a href=&quot;https://developer.mongodb.com/community/forums/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://developer.mongodb.com/community/forums/&lt;/a&gt; sounds more appropriate.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="3682154" author="JIRAUSER1254444" created="Wed, 24 Mar 2021 18:24:17 +0000"  >&lt;p&gt;Hi Ramon,&#160;&lt;/p&gt;

&lt;p&gt;we are running 32k connections on mongodb so facing below error on log file. After suggestions from mongo community to&#160;&lt;b&gt;check pid_max&lt;/b&gt;&#160;and&#160;&lt;b&gt;threads-max&lt;/b&gt;&#160;having little bit high only but number of sockets are opened high which means sockets are not closed yet like below.&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Centos 8.1&lt;/li&gt;
	&lt;li&gt;Mongo version 3.6.17&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;em&gt;&lt;b&gt;cat /proc/sys/kernel/pid_max 4194304&lt;/b&gt;&lt;/em&gt;&lt;br/&gt;
 &lt;em&gt;&lt;b&gt;cat /proc/sys/kernel/threads-max 94465&lt;/b&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;{{mongod 8955 root *366u IPv4 120213606 0t0 TCP testmanager:33445-&amp;gt;node03:49816 (CLOSE_WAIT)&lt;br/&gt;
 mongod 8955 root *367u IPv4 120213789 0t0 TCP testmanager:33445-&amp;gt;node03:49860 (CLOSE_WAIT)&lt;br/&gt;
 mongod 8955 root *368u IPv4 120402126 0t0 TCP testmanager:33445-&amp;gt;node03:49864 (CLOSE_WAIT)&lt;br/&gt;
 mongod 8955 root *369u IPv4 120437763 0t0 TCP testmanager:33445-&amp;gt;node03:49866 (CLOSE_WAIT)}}&lt;/p&gt;

&lt;p&gt;After some time socket descriptors reaching max limit. And mongo throwing an errror for thread creation.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;{{2021-02-24T22:50:04.692+0000 I - &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable&lt;br/&gt;
 2021-02-24T22:50:04.692+0000 W EXECUTOR &lt;span class=&quot;error&quot;&gt;&amp;#91;conn480782&amp;#93;&lt;/span&gt; Terminating session due to error: InternalError: failed to create service entry worker thread&lt;br/&gt;
 2021-02-24T22:50:05.589+0000 I - &lt;span class=&quot;error&quot;&gt;&amp;#91;listener&amp;#93;&lt;/span&gt; pthread_create failed: Resource temporarily unavailable&lt;br/&gt;
 2021-02-24T22:50:05.589+0000 W EXECUTOR &lt;span class=&quot;error&quot;&gt;&amp;#91;conn480783&amp;#93;&lt;/span&gt; Terminating session due to error: InternalError: failed to create service entry worker thread}}&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-17687&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-17687&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Below observation copied from above jira ticket.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;If the issue is not the system-wide limit on the number of threads then the resource exhaustion is somewhere else. You&apos;ll need to investigate what resource is being exhausted (memory and number of file descriptors / sockets are the usual suspects) or simply lower the number of threads. If you&apos;re not using connection pooling you&apos;re probably running out of sockets (netstat -a | grep TIME_WAIT may help).&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;As per analysis, sockets descriptors getting exhausted and mongo thread creating is getting failed. Any suggestions why sockets are not getting closed or any workaround for this.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
 Vasanth&lt;/p&gt;</comment>
                            <comment id="870692" author="ramon.fernandez" created="Wed, 1 Apr 2015 20:26:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dselivanov&quot; class=&quot;user-hover&quot; rel=&quot;dselivanov&quot;&gt;dselivanov&lt;/a&gt;, I see the following in the man page for pthread_create(3):&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;b&gt;EAGAIN&lt;/b&gt;&lt;br/&gt;
Insufficient resources to create another thread, or a system-imposed limit on  the  number  of  threads  was encountered.   The  latter  case  may occur in two ways: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of process for a real user ID, was reached; or the  kernel&apos;s  system-wide limit on the number of threads, /proc/sys/kernel/threads-max, was reached.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;If the issue is not the system-wide limit on the number of threads then the resource exhaustion is somewhere else. You&apos;ll need to investigate what resource is being exhausted (memory and number of file descriptors / sockets are the usual suspects) or simply lower the number of threads. If you&apos;re not using connection pooling you&apos;re probably running out of sockets (&lt;tt&gt;netstat -a | grep TIME_WAIT&lt;/tt&gt; may help).&lt;/p&gt;

&lt;p&gt;I don&apos;t see an evidence of a bug in the server, and since the SERVER project is for reporting bugs or feature suggestions for the MongoDB server I&apos;m going to resolve this issue. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-user group&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="860824" author="dselivanov" created="Mon, 23 Mar 2015 14:27:04 +0000"  >&lt;p&gt;982897&lt;br/&gt;
I believe this is quite big limit?&lt;/p&gt;</comment>
                            <comment id="860758" author="ramon.fernandez" created="Mon, 23 Mar 2015 13:34:35 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dselivanov&quot; class=&quot;user-hover&quot; rel=&quot;dselivanov&quot;&gt;dselivanov&lt;/a&gt;, there&apos;s an error message in the log that says:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;pthread_create: Resource temporarily unavailable&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;You may need to lower the number of threads, or increase some system limits. I think the error above means &lt;tt&gt;pthread_create&lt;/tt&gt; returned EAGAIN; can you check what&apos;s the maximum number of threads allowed on this system?&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$ cat /proc/sys/kernel/threads-max&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="68385" name="mongodb_error_log" size="17318" author="dselivanov" created="Mon, 23 Mar 2015 11:49:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 23 Mar 2015 13:27:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 46 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 46 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dselivanov</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>vasanth3g@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrla73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs7dav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsg6f3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>