<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:52:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-644] SocketException causes cleanup of the entire DBPortPool leading to OutOfMemoryErrors in high load applications</title>
                <link>https://jira.mongodb.org/browse/JAVA-644</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;I have an application under high load connected to a MongoDB (2.0.7) using the Java Driver. When the DB fails to allocate memory (mmap), the connection on the client side is reset. The SocketException thrown is causing a cleanup of the entire pool of socket connections in the DBPortPool implementation. Is there a reason for that?&lt;/p&gt;

&lt;p&gt;A cleanup of the entire pool leads to OutOfMemoryErrors since connection (de)allocation is expensive. A failure of a single DB operation can affect an entire client application.&lt;/p&gt;

&lt;p&gt;Would it not be better to clean up just the connection that was reset?&lt;/p&gt;

&lt;p&gt;Thanks&lt;/p&gt;</description>
                <environment>Oracle JVM, Linux 64bit</environment>
        <key id="50744">JAVA-644</key>
            <summary>SocketException causes cleanup of the entire DBPortPool leading to OutOfMemoryErrors in high load applications</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="robertgacki">Robert Gacki</reporter>
                        <labels>
                    </labels>
                <created>Tue, 18 Sep 2012 09:16:49 +0000</created>
                <updated>Wed, 4 Dec 2013 18:47:34 +0000</updated>
                            <resolved>Wed, 4 Dec 2013 18:47:34 +0000</resolved>
                                    <version>2.8.0</version>
                    <version>2.9.1</version>
                                                    <component>Connection Management</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="465629" author="jeff.yemin" created="Wed, 4 Dec 2013 18:47:34 +0000"  >&lt;p&gt;Thanks for the reply.  Closing.&lt;/p&gt;


&lt;p&gt;Jeff&lt;/p&gt;</comment>
                            <comment id="465628" author="robertgacki" created="Wed, 4 Dec 2013 18:46:56 +0000"  >&lt;p&gt;Hi Jeff, &lt;br/&gt;
no, I did no further load tests to reproduce the issue. And we did not see the issue in production. You can close the ticket.&lt;/p&gt;</comment>
                            <comment id="465497" author="jeff.yemin" created="Wed, 4 Dec 2013 16:13:24 +0000"  >&lt;p&gt;Hi Robert,&lt;/p&gt;

&lt;p&gt;Do you have any updates?  If not, I&apos;m going to close this ticket.&lt;/p&gt;

</comment>
                            <comment id="384954" author="robertgacki" created="Fri, 19 Jul 2013 19:38:23 +0000"  >&lt;p&gt;Hi Jeff,&lt;/p&gt;

&lt;p&gt;I&apos;m not sure either. But it was reproducible at that time. I suspect the overhead to create new connections. Does the driver exchange data first, when connected? Maybe it&apos;s the buffers filling up all at once, when the HTTP requests hit the application.&lt;/p&gt;

&lt;p&gt;Anyway, my project planned to do load tests again. I will bring that up again so we can analyse it further.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Robert&lt;/p&gt;</comment>
                            <comment id="380002" author="jeff.yemin" created="Fri, 12 Jul 2013 18:34:30 +0000"  >&lt;p&gt;Robert,&lt;/p&gt;

&lt;p&gt;Can you explain how controlling the rate of connection allocations will result in there being any less live objects on the heap in the end.  It seems to me that ultimately we will end up with the same live heap, and you either have enough memory to hold it or you don&apos;t.  &lt;/p&gt;</comment>
                            <comment id="367295" author="robertgacki" created="Tue, 25 Jun 2013 14:44:42 +0000"  >&lt;p&gt;1. Xmx was set to 3g.&lt;br/&gt;
2. I&apos;m not sure anymore. But I think we had between 500 and up to 1.000 connections per host. Our current configurations use 500, though.&lt;/p&gt;

&lt;p&gt;A common solution is to allow the pool to warm up by configuring a minimum pool size. The pool will then allocate new connections at a controlled rate (in a separate thread with a configurable rate) instead of being populated by a herd of requests. The pool will become available, when the allocation is finished. In my case, I could set that minimum size to match the pool&apos;s maximum size. It&apos;s better for me to wait seconds longer for the application to become available then to have it fail by an OOM and to have the JVM restarted.&lt;/p&gt;</comment>
                            <comment id="367233" author="jeff.yemin" created="Tue, 25 Jun 2013 13:12:26 +0000"  >&lt;p&gt;A few questions:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;What have you set -Xmx to for java?&lt;/li&gt;
	&lt;li&gt;What have you set connectionsPerHost to in MongoClientOptions?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;If the primary goes down all the connections will be dropped by the server, so the connection pool needs to be cleared in any case.  Do you have an alternative to suggest?&lt;/p&gt;
</comment>
                            <comment id="367223" author="robertgacki" created="Tue, 25 Jun 2013 12:55:57 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;at that time, we did load tests. And in this scenario, we simulated an outtage of the Master. When that happened, the #gotError method of the DBPortPool class received either a SocketException or a EOFException (BSON) and all connections of the pool were closed. The OOM occured, when the load was still high and the pool was repopulated with new connections after the (new) Master became available, again.&lt;/p&gt;

&lt;p&gt;From my POV, the driver should not cause OOMs when there is a connectivity problem, even under high load, because fail-over is a feature of the driver. So I questioned the strategy of dumping the entire pool.&lt;/p&gt;

&lt;p&gt;Of course, if there is another way to mitigate the problem, I&apos;d appreciate any hints / best practices.&lt;/p&gt;

&lt;p&gt;Best,&lt;br/&gt;
Robert&lt;/p&gt;</comment>
                            <comment id="367006" author="jeff.yemin" created="Tue, 25 Jun 2013 04:19:09 +0000"  >&lt;p&gt;It&apos;s not clear to me how closing a connection, even if an expensive operation, would lead to OOM.  Can you elaborate?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrhswn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>17494</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>