<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:56:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-2169] ConcurrentPool writes to same connection buffer then keeps waiting</title>
                <link>https://jira.mongodb.org/browse/JAVA-2169</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;This would happen only when most/all writes are unacknowledged.&lt;/p&gt;

&lt;p&gt;Each write happens to last available connection based on code in:&lt;br/&gt;
mongo-java-driver-master/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java:130&lt;br/&gt;
`T t = available.pollLast();`&lt;/p&gt;

&lt;p&gt;Each release of connection again appends back the available connection to end&lt;br/&gt;
/goshposh/mongo-java-driver-master/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java:84&lt;br/&gt;
`available.addLast(t);`&lt;/p&gt;

&lt;p&gt;Situation:&lt;br/&gt;
Trying to use mongo java driver in jruby, seeing lower performance than ruby driver.&lt;br/&gt;
We see that `collection.updateOne` takes about 20 seconds on client but mongo server logs have only 100ms as the time taken for query.&lt;br/&gt;
JavaStack reveals that thread spend most of the time in `com.mongodb.connection.SocketStream.write`&lt;/p&gt;

&lt;p&gt;Logical explanation:&lt;br/&gt;
Assume Mongo is takes 10 seconds to execute a query.&lt;br/&gt;
You have 2 threads and 100 connection pool size. &lt;/p&gt;

&lt;p&gt;Thread 1 writes to 100th connection buffer - fire &amp;amp; forget&lt;br/&gt;
Thread 2 writes to 99th connection buffer - fire &amp;amp; forget&lt;br/&gt;
thread 1 completes writing to buffer ( mongo::unacknowledged), releases back the connection to pool at the last.&lt;br/&gt;
thread 2 completes writing to buffer ( mongo::unacknowledged), releases back the connection to pool at the 100.&lt;/p&gt;

&lt;p&gt;Now thread 1 again makes a write, this time it again picks the last buffer ( originally 99th). This TCP send buffer already had something that had to be picked for mongo, but we wrote into the same buffer since it was available to write.&lt;br/&gt;
Now thread 2 will write to 100th buffer , which is again non-empty. &lt;/p&gt;

&lt;p&gt;Eventually there will be a state where buffer are full and mongo client will just wait on java.net.SocketOutputStream.socketWrite0.&lt;/p&gt;

&lt;p&gt;Although the buffer 1 to 98 are empty it will not be used since connection is available on 99 &amp;amp; 100 ( but  the TCP Send buffer is full )&lt;/p&gt;


&lt;p&gt;Trace attached.&lt;/p&gt;

&lt;p&gt;Ruby driver ( we use 1.8.5) doesn&apos;t have this problem since it does a random on connection:&lt;br/&gt;
mongo-ruby-driver-1.8.5/lib/mongo/util/pool.rb&lt;br/&gt;
socket = available&lt;span class=&quot;error&quot;&gt;&amp;#91;rand(available.length)&amp;#93;&lt;/span&gt;&lt;/p&gt;
</description>
                <environment>ubuntu 12</environment>
        <key id="278163">JAVA-2169</key>
            <summary>ConcurrentPool writes to same connection buffer then keeps waiting</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="gauravshah">Gaurav Shah</reporter>
                        <labels>
                    </labels>
                <created>Thu, 7 Apr 2016 12:23:34 +0000</created>
                <updated>Wed, 11 Sep 2019 19:13:43 +0000</updated>
                            <resolved>Sun, 2 Oct 2016 22:15:39 +0000</resolved>
                                    <version>3.2.2</version>
                                                    <component>Connection Management</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1231092" author="gauravshah" created="Sat, 9 Apr 2016 06:37:44 +0000"  >&lt;p&gt;True, I do not see a way to check the state of buffer from java. Currently I have set minConnectionPerHost and applied a patch that releases connection to the pool at the beginning ( instead of end). This solves the problem for now. Will send in a pull request after testing it for couple of days.&lt;/p&gt;</comment>
                            <comment id="1230550" author="jeff.yemin" created="Fri, 8 Apr 2016 18:01:05 +0000"  >&lt;p&gt;Hi Gaurav,&lt;/p&gt;

&lt;p&gt;I didn&apos;t mean to suggest that the proposal does not have merit, only that there may be a workaround that is acceptable.  Given that the default stack size is 1MB for 64-bit JVMs, you could scale up the number of threads for what I would consider a modest amount of extra memory.&lt;/p&gt;

&lt;p&gt;In your last response, you mentioned that the driver could check the state of the send buffer.  I don&apos;t see a way to do that via Java&apos;s Socket API.  Is there a technique that you&apos;re aware of that would allow the driver to do that?&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Jeff&lt;/p&gt;</comment>
                            <comment id="1230478" author="gauravshah" created="Fri, 8 Apr 2016 17:13:13 +0000"  >&lt;p&gt;sorry for the confusion, the high CPU is on the old code, which means we were getting enough throughput.&lt;/p&gt;

&lt;p&gt;On the new code as you told the CPU is low, but if I increase the application threads I would be wasting memory. It feels wrong to increase threads for this reason. The driver could be more smart by checking the state of buffer if its more than 50% full and then move to next connection&lt;/p&gt;</comment>
                            <comment id="1230411" author="jeff.yemin" created="Fri, 8 Apr 2016 16:09:29 +0000"  >&lt;p&gt;If both threads are blocking on java.net.SocketOutputStream.socketWrite0, then CPU usage should be quite low.  How are you measuring CPU usage for the application?&lt;/p&gt;</comment>
                            <comment id="1230091" author="gauravshah" created="Fri, 8 Apr 2016 11:23:15 +0000"  >&lt;p&gt;We have 20 machines each running two threads. The machines are already at spike on CPU, so cannot increase threads. If we add more machines , we would be wasting money on it, since the throughput that we are getting is good enough&lt;/p&gt;</comment>
                            <comment id="1230084" author="gauravshah" created="Fri, 8 Apr 2016 11:18:45 +0000"  >&lt;p&gt;checking the TCP buffer if its full will also not help since tcp buffer could have capacity of 4096 but the buffer might be filled only till 4080 and the next bytes to be written are 100 . &lt;/p&gt;</comment>
                            <comment id="1230081" author="gauravshah" created="Fri, 8 Apr 2016 11:15:50 +0000"  >&lt;p&gt;the application is CPU &amp;amp; mongo intensive with two threads, if I add in more threads it might not work well on CPU end of the application.&lt;/p&gt;</comment>
                            <comment id="1230075" author="jeff.yemin" created="Fri, 8 Apr 2016 11:02:11 +0000"  >&lt;p&gt;I see.  What about using multiple threads to spread the write load to more sockets?  &lt;/p&gt;</comment>
                            <comment id="1230072" author="gauravshah" created="Fri, 8 Apr 2016 10:54:57 +0000"  >&lt;p&gt;Hi Jeff,&lt;/p&gt;

&lt;p&gt;The complete application uses only unacknowledged writes&lt;/p&gt;</comment>
                            <comment id="1230070" author="jeff.yemin" created="Fri, 8 Apr 2016 10:52:51 +0000"  >&lt;p&gt;Hi Gaurav,&lt;/p&gt;

&lt;p&gt;Have you considered using a dedicated MongoClient instance for the application&apos;s unacknowledged writes?  &lt;/p&gt;</comment>
                            <comment id="1229871" author="gauravshah" created="Fri, 8 Apr 2016 04:33:24 +0000"  >&lt;p&gt;changing the order of release will still not help since the new connections are not even open. So although the max_pool size is 100. The current pool size will only be 2, Not sure how to get around it.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="116777" name="mongo-java-driver trace.txt" size="2264" author="gauravshah" created="Thu, 7 Apr 2016 12:23:34 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsk75r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>