<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 09:02:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[JAVA-4452] Improve performance characteristics of the connection pool to be on par with the driver 4.2.0</title>
                <link>https://jira.mongodb.org/browse/JAVA-4452</link>
                <project id="10006" key="JAVA">Java Driver</project>
                    <description>&lt;p&gt;We observed a significant regression in the ycsb_100read benchmark (see &lt;a href=&quot;https://github.com/mongodb-labs/YCSB/blob/03515f78fc460836886604ff8fbee89e5853fbbc/ycsb-mongodb/workloads/workloadc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ycsb-mongodb/workloads/workloadc&lt;/a&gt; and &lt;a href=&quot;https://github.com/10gen/dsi/blob/4fc1d9303cf9d22050c3ef168ecb8370e0e966a4/configurations/test_control/test_control.ycsb.yml#L27-L51&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;dsi/configurations/test_control/test_control.ycsb.yml&lt;/a&gt;). Just to give a perspective, the throughput dropped from 55k for 4.2.0, to 36k for 4.3.0, to 19k for 4.4.0 when the ycsb_100read benchmark is run against a single-node MongoDB replica set.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1970791">JAVA-4452</key>
            <summary>Improve performance characteristics of the connection pool to be on par with the driver 4.2.0</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="valentin.kovalenko@mongodb.com">Valentin Kavalenka</assignee>
                                    <reporter username="valentin.kovalenko@mongodb.com">Valentin Kavalenka</reporter>
                        <labels>
                    </labels>
                <created>Tue, 25 Jan 2022 21:26:52 +0000</created>
                <updated>Sat, 28 Oct 2023 11:20:52 +0000</updated>
                            <resolved>Thu, 27 Jan 2022 16:17:51 +0000</resolved>
                                                    <fixVersion>4.5.0</fixVersion>
                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="4318592" author="xgen-internal-githook" created="Thu, 27 Jan 2022 16:17:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Valentin Kovalenko&apos;, &apos;email&apos;: &apos;valentin.male.kovalenko@gmail.com&apos;, &apos;username&apos;: &apos;stIncMale&apos;}
&lt;p&gt;Message: Alternate fair/unfair lock modes in the connection pool to improve throughput (#862)&lt;/p&gt;

&lt;p&gt;Another tiny change is moving `pool.release(openConnection)` out of the guarded section&lt;br/&gt;
in `DefaultConnectionPool.OpenConcurrencyLimiter.tryHandOverOrRelease`:&lt;br/&gt;
it does not have to be guarded, and when not only fair locking is used,&lt;br/&gt;
this change has a significant positive measurable effect&lt;br/&gt;
on the connection cycle (check out/check in) throughput.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-4452&quot; title=&quot;Improve performance characteristics of the connection pool to be on par with the driver 4.2.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-4452&quot;&gt;&lt;del&gt;JAVA-4452&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/commit/146e7ce7e9b4839ba0959f45116d298c713c5abb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-java-driver/commit/146e7ce7e9b4839ba0959f45116d298c713c5abb&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4316187" author="JIRAUSER1258163" created="Wed, 26 Jan 2022 18:24:50 +0000"  >&lt;h2&gt;&lt;a name=&quot;Thecause&quot;&gt;&lt;/a&gt;The cause&lt;/h2&gt;
&lt;p&gt;The degradation is caused by &lt;b&gt;fair&lt;/b&gt; locking introduced at first by &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-3927&quot; title=&quot;Rate limit new connection creations (maxConnecting)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-3927&quot;&gt;&lt;del&gt;JAVA-3927&lt;/del&gt;&lt;/a&gt; in 4.3.0 (&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/blob/fb5bc205515a61927ddbc95b65fe7474a5f4b868/driver-core/src/main/com/mongodb/internal/connection/DefaultConnectionPool.java#L841&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;DefaultConnectionPool.OpenConcurrencyLimiter.lock&lt;/tt&gt;&lt;/a&gt;) and then additionally by &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-3928&quot; title=&quot;Connection pool paused state &quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-3928&quot;&gt;&lt;del&gt;JAVA-3928&lt;/del&gt;&lt;/a&gt; in 4.4.0 (&lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/blob/fb5bc205515a61927ddbc95b65fe7474a5f4b868/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java#L344&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;ConcurrentPool.StateAndPermits.lock&lt;/tt&gt;&lt;/a&gt;). This was a hunch, which I am calling &quot;an educated guess&quot; to sound like other grown ups. To test this hypothesis, I run ycsb_100read via the &lt;a href=&quot;https://github.com/10gen/dsi&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;DSI&lt;/a&gt; against each of the following: 4.2.0, snapshot of 4.5.0, modified 4.5.0 that uses unfair locks in the connection pool. The results showed that with unfair locking 4.5.0-snapshot is on par with 4.2.0 (values represent the throughput in ops/s against a standalone MongoDB installation which is the only topology I run ycsb_100read against):&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;workload threads \ driver version&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.2.0&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.5.0-snapshot&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.5.0-snapshot-unfair&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1000&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;44648&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;15595&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;42399&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;100&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;56489&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;18258&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;57612&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;I compared only the snapshot of 4.5.0 (specifically, &lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/tree/c52ff80852179be7cd4c9e3df398acb3ba7fa6f1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;c52ff80&lt;/a&gt;) with 4.2.0, disregarding 4.3.0 and 4.4.0 because the same regression is observable with the snapshot of 4.5.0.&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Thebaseline&quot;&gt;&lt;/a&gt;The baseline&lt;/h2&gt;
&lt;p&gt;The cause is clear, now we need to gather specific characteristics of 4.2.0 to treat them as the baseline when judging the fix. Having the ycsb_100read benchmark is good, but I also wanted to have a more narrow benchmark that measures performance characteristics of a connection cycle (check out / check in). For this purpose I introduced this &lt;a href=&quot;https://github.com/stIncMale/mongo-java-driver/blob/f0fe198907ac22dcb1a8818432d79c77b30ef9d5/src/main/java/com/mongodb/DefaultConnectionPoolBenchmark.java#L223-L231&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;connectionCycle&lt;/a&gt; benchmark written with JMH.&lt;/p&gt;

&lt;p&gt;When viewing the results, one should keep in mind the following&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;ycsb_100read always has &lt;a href=&quot;https://github.com/mongodb-labs/YCSB/blob/03515f78fc460836886604ff8fbee89e5853fbbc/ycsb-mongodb/mongodb/src/main/java/com/yahoo/ycsb/db/MongoDbClient.java#L268&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;enough connections in the pool to serve all workload threads&lt;/a&gt;, while connectionCycle is sometimes run with less connections that workload threads.&lt;/li&gt;
	&lt;li&gt;ycsb_100read was run on a &lt;a href=&quot;https://ark.intel.com/content/www/us/en/ark/products/75277/intel-xeon-processor-e52680-v2-25m-cache-2-80-ghz.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;computer with 10 cores, 20 threads&lt;/a&gt;, while connectionCycle was run on a &lt;a href=&quot;https://www.intel.com/content/www/us/en/products/sku/191045/intel-core-i79750h-processor-12m-cache-up-to-4-50-ghz/specifications.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;computer with 6 cores, 12 threads&lt;/a&gt;. So the results are not right away comparable across the benchmarks.&lt;/li&gt;
	&lt;li&gt;p0, p0.5, etc. are percentiles.&lt;/li&gt;
	&lt;li&gt;For some reason, the runner for ycsb_100read does not calculate p0.95 and p0.99 latencies for 1000 workload threads.&lt;/li&gt;
	&lt;li&gt;Throughput units: ops/s for ycsb_100read, ops/ms for connectionCycle.&lt;/li&gt;
	&lt;li&gt;Latency units: &#956;s for ycsb_100read, ms for connectionCycle.&lt;/li&gt;
	&lt;li&gt;The &quot;&#177;&quot; part represents 99.9% confidence interval, assuming that the distribution of all measurements is Gaussian.&lt;/li&gt;
	&lt;li&gt;All latency distributions were measured with the system under test being severely loaded to achieve maximum throughput. These latencies represent the worse case, and were used only to assess the fairness characteristics.&lt;/li&gt;
&lt;/ul&gt;


&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;ycsb_100read, 4.2.0&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;metric \ workload threads&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;100&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;6&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;throughput, ops/s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;44648&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;56489&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;17172&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;3066&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;avg latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;21228&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1748&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;344&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;321&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0 (min) latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;266&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;260&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;270&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;295&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.95 latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2140&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;380&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;340&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.99 latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;3510&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;410&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p1 (max) latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;6755862&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;958470&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;363509&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;328838&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;connectionCycle, 4.2.0&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;metric \ workload threads, max pool size&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 1000&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;6, 6&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1, 1&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 250&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;throughput, ops/ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2687.579 &#177; 91.716&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2730.714 &#177; 31.349&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;5081.488 &#177; 26.004&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;33.639 &#177; 1.238&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;avg latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.354 &#177; 0.003&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;29.480 &#177; 0.002&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0 (min) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#8308;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;27.984&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.5 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.004&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;29.295&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.008&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;29.983&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.95 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.009&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;30.671&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.99 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.014&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;34.603&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.025&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;41.091&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;886.047&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;42.598&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p1 (max) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2071.986&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;42.795&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;Some of the baseline metric values deserve explanations.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;ThebaselineconnectionCyclethroughputfor%281000threads%2C250connections%29isinteresting&quot;&gt;&lt;/a&gt;The baseline connectionCycle throughput for (1000 threads, 250 connections) is interesting&lt;/h3&gt;
&lt;p&gt;34 ops/ms is 34_000 ops/s, i.e., just checking a connection out and checking it back has smaller throughput than the throughput in ycsb_100read for 1000 threads, 1000 connections, which is 45_000 ops/s. As I mentioned, we can&apos;t directly compare the results, because they were obtained on different computers, but we definitely can tell that the throughput measured in connectionCycle for this scenario is very low. I am sure ycsb_100read will notice this for 4.2.0 if run with fewer connections than the number of workload threads.&lt;/p&gt;

&lt;p&gt;The reason behind is that 4.2.0 uses a fair semaphore to enforce pool&apos;s &lt;tt&gt;maxSize&lt;/tt&gt;. When there are enough available permits for all threads, threads do not queue waiting for an available permit, and fairness of the semaphore plays no role. However, when there are not enough available permits, which is definitely the case for (1000 threads, 250 connections), threads are not only queued waiting for a permit, but they all have to go at the end of the queue and wait their turn because the semaphore is fair. Due to this behavior, the throughput in connectionCycle drops by an order of magnitude. However, the latencies are all very similar, ranging from 28 to 43 ms.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;ThebaselineconnectionCyclehighpercentilesoflatenciesfor%281000threads%2C1000connections%29areinteresting&quot;&gt;&lt;/a&gt;The baseline connectionCycle high percentiles of latencies for (1000 threads, 1000 connections) are interesting&lt;/h3&gt;
&lt;p&gt;p0.9999 is almost 1 s, p1 is about 2 s. Remember, the benchmark just checks out and checks in connections, not doing anything else, and yet sometimes it takes 2 seconds. It is interesting and surprising at first because the semaphore used is fair. However, as explained above, threads are not blocked by this semaphore because there are enough available permits for them, which means that semaphore&apos;s fairness does not play a role. Instead, given that 1000 threads is way more than the number of threads the CPU can run in parallel, we likely observe here the unfairness of the OS thread scheduler. Thanks, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jeff.yemin&quot; class=&quot;user-hover&quot; rel=&quot;jeff.yemin&quot;&gt;jeff.yemin&lt;/a&gt;, for the idea of this explanation (my initial explanation was proven to be incorrect).&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Goingdeeperintothecause&quot;&gt;&lt;/a&gt;Going deeper into the cause&lt;/h2&gt;
&lt;p&gt;We saw that making blocking behavior unfair in 4.5.0 solves the regression, but why is that the case, given that the connection pool in 4.2.0 also blocks threads in a fair manner by using a fair semaphore?&lt;/p&gt;

&lt;p&gt;Implementing each of the aforementioned tasks &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-3927&quot; title=&quot;Rate limit new connection creations (maxConnecting)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-3927&quot;&gt;&lt;del&gt;JAVA-3927&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/JAVA-3928&quot; title=&quot;Connection pool paused state &quot; class=&quot;issue-link&quot; data-issue-key=&quot;JAVA-3928&quot;&gt;&lt;del&gt;JAVA-3928&lt;/del&gt;&lt;/a&gt; requires signalling to blocked threads when the condition they are waiting for happens, and doing this in a way that makes losing the signal impossible. Within the concurrency primitives known to me, condition variable (&lt;a href=&quot;https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/locks/Condition.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;java.util.concurrent.locks.Condition&lt;/tt&gt;&lt;/a&gt;) is the only way to do this. As a result, I had to replace the semaphore with a home-grown lock-based semaphore (&lt;tt&gt;ConcurrentPool.StateAndPermits&lt;/tt&gt;), because this way I have access to the &lt;tt&gt;Condition&lt;/tt&gt; API. A significant difference between &lt;a href=&quot;https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/concurrent/Semaphore.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;java.util.concurrent.Semaphore&lt;/tt&gt;&lt;/a&gt; and &lt;tt&gt;ConcurrentPool.StateAndPermits&lt;/tt&gt; from the performance perspective is that they have different progress guarantees in a situation when there are enough available permits: &lt;tt&gt;ConcurrentPool.StateAndPermits.acquirePermit&lt;/tt&gt;/&lt;tt&gt;releasePermit&lt;/tt&gt; is blocking, while &lt;tt&gt;Semaphore.acquire&lt;/tt&gt;/&lt;tt&gt;release&lt;/tt&gt; is lock-free&lt;sup&gt;1&lt;/sup&gt;, but not wait-free: compare this &lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/blob/fb5bc205515a61927ddbc95b65fe7474a5f4b868/driver-core/src/main/com/mongodb/internal/connection/ConcurrentPool.java#L378-L385&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;code in &lt;tt&gt;ConcurrentPool.StateAndPermits.acquirePermit&lt;/tt&gt;&lt;/a&gt; with &lt;a href=&quot;https://github.com/openjdk/jdk/blob/e3076552ec528864e61a6e0ec91e228006fddefc/src/java.base/share/classes/java/util/concurrent/Semaphore.java#L249-L259&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;code in Semaphore&lt;/a&gt;, which represents a happy path for &lt;tt&gt;Semaphore.acquire&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;It is now clear that in ycsb_100read (1000 threads, 1000 connections) workload threads are not blocked in 4.2.0, but are blocked in 4.5.0 (even though no thread waits for an available permit because there are enough of them), and blocked in a fair manner, which further reduces the throughput measured by the benchmark.&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Asolution%3Agetridoffairness&quot;&gt;&lt;/a&gt;&lt;del&gt;A solution&lt;/del&gt;: get rid of fairness&lt;/h2&gt;
&lt;p&gt;Wishful thinking led me to the following hypothesis: maybe we don&apos;t actually need fairness? I modified 4.2.0 to use an unfair semaphore, and took a look at the latency distribution in the connectionCycle benchmark. The results are&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;connectionCycle, 4.2.0-unfair&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;metric \ workload threads, max pool size&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 1000&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;6, 6&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1, 1&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 250&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;throughput, ops/ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2673.850 &#177; 166.890&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2729.713 &#177; 60.227&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;5130.801 &#177; 116.675&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2754.849 &#177; 172.705&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;avg latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.358 &#177; 0.004&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.383 &#177; 0.003&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0 (min) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#8308;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#8309;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.5 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.004&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.008&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.007&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.95 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.009&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.009&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.99 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.014&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.014&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.027&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;73.138&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;917.504&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;590.348&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p1 (max) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2048.918&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2122.318&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;We can see that unfair locking which comes into play for (1000 threads, 250 connections) causes quite significant latencies. This seems to be the evidence in support of the &lt;a href=&quot;https://github.com/mongodb/specifications/blob/568093ce7f0e1394cf4952c417e1e7dacc5fef53/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#waitqueue&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;connection pool specification fairness requirement&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Theproposedsolution%3Aalternatebetweenfairandunfairmodes&quot;&gt;&lt;/a&gt;The proposed solution: alternate between fair and unfair modes&lt;/h2&gt;
&lt;p&gt;What if we had fair locking behavior only when there are not enough available permits, and unfair locking behavior when there are enough of them? This behavior is what we had in 4.2.0 with semaphore, provided that only fairness is taken into account.&lt;/p&gt;

&lt;p&gt;Fairness of Java SE API locks is not re-configurable. However, &lt;tt&gt;ReentrantReadWriteLock.WriteLock.tryLock&lt;/tt&gt; disregards fairness and allows a thread to acquire a lock while ignoring all the queued threads, provided that the lock is not held by another thread. Calling &lt;tt&gt;tryLock&lt;/tt&gt; and then &lt;tt&gt;lock&lt;/tt&gt; if the former did not succeed, emulates unfair lock to an extent.&lt;/p&gt;

&lt;p&gt;All we need now is to know whether there are waiting threads in order to decide if fair locking is needed. &lt;tt&gt;ReentrantReadWriteLock.hasWaiters&lt;/tt&gt; requires the calling thread to hold the lock, and we need to know whether there are waiters before acquiring the lock. Well, we can estimate this by counting waiters&lt;sup&gt;2&lt;/sup&gt; on our own.&lt;/p&gt;

&lt;p&gt;Here is what we get as a result of applying this approach:&lt;/p&gt;

&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;ycsb_100read, 4.5.0-snapshot-proposed&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;metric \ workload threads&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;100&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;6&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;throughput, ops/s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;47704&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;57610&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;16358&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2887&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;avg latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;20805&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1709&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;362&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;341&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0 (min) latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;257&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;276&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;289&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;304&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.95 latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1960&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;400&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;380&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.99 latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;3000&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;440&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;480&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p1 (max) latency, &#956;s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1985727&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;564845&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;389947&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;351269&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;connectionCycle, 4.5.0-snapshot-proposed&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;metric \ workload threads, max pool size&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 1000&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;6, 6&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1, 1&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;1000, 250&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;throughput, ops/ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;656.229 &#177; 181.564&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2066.979 &#177; 52.481&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;3207.597 &#177; 183.752&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;179.161 &#177; 60.814&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;avg latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.967 &#177; 0.002&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;4.454 &#177; 0.010&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0 (min) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#8308;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#8308;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.5 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#179;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#8776; 10&#8315;&#179;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.001&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;28.574&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.95 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;0.001&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;31.457&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.99 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;31.556&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;69.075&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;45.285&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;77.070&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p0.9999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;65.864&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;93.061&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;p1 (max) latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;120.717&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;126.616&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;Let&apos;s compare this with the baseline and 4.5.0-snapshot:&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;benchmark; metric \ version&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.2.0 (baseline)&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.5.0-snapshot&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;4.5.0-snapshot-proposed&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;comment&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;ycsb_100read, (1000 threads, 1000 connections); throughput, ops/s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;45k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;48k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128077; Obviously, the throughout here is not better, it is just not worse, which is good.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 1000 connections); throughput, ops/s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2688k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;33k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;656k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128078;&#128077; Not good, but apparently, enough to achieve the baseline throughput in an integration benchmark such as ycsb_100read. That is why I consider this acceptable. Also, it&apos;s a significant improvement comparing to 4.5.0-snapshot.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 1000 connections); p0.9999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;886&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;66&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128077;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 1000 connections); p1 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2072&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;121&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128077;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 250 connections); throughput, ops/s&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;34k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;179k&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128077;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 250 connections); p0.9999 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;43&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;93&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128078;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;connectionCycle, (1000 threads, 250 connections); p1 latency, ms&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;43&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;-&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;127&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#128078;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;I don&apos;t think that the &lt;a href=&quot;https://github.com/mongodb/mongo-java-driver/pull/862&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;proposed change&lt;/a&gt; is great, but it appears to be good enough to resolve the regression. Maybe with higher number of workload threads in the ycsb_100read benchmark or other integration benchmarks, the regression still manifests itself (though, definitely to a lower extent). Unfortunately, when I tried running ycsb_100read with 5000 threads, the benchmark runner failed to stop the benchmark after &lt;tt&gt;maxexecutiontime&lt;/tt&gt;, and I did not try to play around with configuration or figure out the reason, I simply stopped at 1000 threads.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;P.S.&lt;/b&gt; It&apos;s a long report, but I want the show the reasoning in detail, as well as provide other details because it opens opportunities for other engineers to find problems in the reasoning, question it, and either prove me wrong, or maybe even propose a better approach.&lt;/p&gt;
&lt;hr /&gt;
&lt;p&gt;&lt;sup&gt;1&lt;/sup&gt; Clearly, a semaphore itself is a blocking synchronization mechanism, but we are interested specifically in the progress guarantees of acquiring a permit when the max number of permits all threads that use the semaphore may ever want to hold concurrently is not larger than the number of permits the semaphore maintains.&lt;br/&gt;
&lt;sup&gt;2&lt;/sup&gt; It may seem viable to use &lt;tt&gt;permits&lt;/tt&gt; &amp;gt; 0 as a way to decide that there are likely no waiters, but benchmarking shows that with this approach high percentiles of contended connectionCycle latencies (when the number of threads that use the pool is higher than the maximum pool size) become similar to a situation when no fair locking is used. That is, this approach does not result in the behavior we want.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1970663">JAVA-4451</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="1974218">JAVA-4462</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is caused by">
                                        <issuelink>
            <issuekey id="1585341">JAVA-3927</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1585347">JAVA-3928</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1971802">JAVA-4454</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10257" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Documentation Changes</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="11861"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i014qf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>