<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 21:43:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[CSHARP-2689] Occasional extremely poor performance when connection pool is exhausted</title>
                <link>https://jira.mongodb.org/browse/CSHARP-2689</link>
                <project id="10041" key="CSHARP">C# Driver</project>
                    <description>&lt;h1&gt;&lt;a name=&quot;Issue&quot;&gt;&lt;/a&gt;Issue&lt;/h1&gt;

&lt;p&gt;We use use the C# MongoDB driver in an ASP .NET Core application and experience occasionally extremely poor performance. Some requests take 80-90 seconds, where they normally take ~1 seconds. This problem is triggered there is issued a large number of parallel tasks which fetches data from MongoDB. The data is fetched using the Async API.&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;Repro%26TestResults&quot;&gt;&lt;/a&gt;Repro &amp;amp; Test Results&lt;/h1&gt;

&lt;p&gt;I&apos;ve put together a minimal repro example illustrating the problem. The Visual Studio solution is attached on this ticket.&lt;/p&gt;

&lt;p&gt;The issue occurs when the following is run:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;async Task Fetch()&lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: {
&#160; await collection.Find(filter).ToListAsync();
&#160; await collection.Aggregate&amp;lt;BsonDocument&amp;gt;(pipeline).ToListAsync();
}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;var tasks = Enumerable.Range(0, count).Select(i =&amp;gt; Fetch()).ToList();&lt;br/&gt;
await Task.WhenAll(tasks);&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The filter is simply for a single _id and the pipeline is a match for the same _id. The collection contains a single document with the _id requested. The &apos;count&apos; variable is used to test various number in parallel. Strangely the problem goes away if I swap the order of the Find &amp;amp; Aggregate queries, or comment out one of them.&lt;/p&gt;

&lt;p&gt;When this code is run it starts to run extremely slow with a count of ~200-250 tasks. However it appears to run slow only the first time. Here is some run times with various counts, each time time the application is restarted and the queries are run two times:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Run 1. Fetched 150 in 606 ms&lt;br/&gt;
Run 2. Fetched 150 in 192 ms&lt;/p&gt;

&lt;p&gt;Run 1. Fetched 200 in 46&#160;801 ms&lt;br/&gt;
Run 2. Fetched 200 in 606 ms&lt;/p&gt;

&lt;p&gt;Run 1. Fetched 250 in 68&#160;233 ms&lt;br/&gt;
Run 2. Fetched 250 in 1&#160;667 ms&lt;/p&gt;

&lt;p&gt;Run 1. Fetched 20 in 359 ms&lt;br/&gt;
Run 2. Fetched 2500 in 2&#160;843 ms&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;At ~200 the first parallel queries take over 46 seconds. However if the first query has few in parallel, the second handles a lot more (Like the last example above, where running 2500 i parallel finishes in only 2.8 seconds). When the queries run fast (the first example above), I see in Windows Task Manager the number of threads for the application staying around ~30, when the problem occurs the number of threads increases by 1 for every second. So to me, when the problem occurs it appears that even though I&apos;m using the Async methods, something is blocking causing new threads to constantly be spawned.&lt;/p&gt;

&lt;p&gt;If the problem only would occur at the first request it would not be such a big deal, however in&#160;our ASP .NET Core application we see the problem not only for the first request, but occasionally (The IMongoClient &amp;amp; IMongoDatabase are singletons).&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;TemporaryFix&quot;&gt;&lt;/a&gt;Temporary Fix&lt;/h1&gt;

&lt;p&gt;Increasing the MaxConnectionPool of the MongoClient to f.ex. 1000 makes the problem disappear. However I think this is a band-aid fix, just masking the problem.&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;Sidenote&quot;&gt;&lt;/a&gt;Sidenote&lt;/h1&gt;

&lt;p&gt;I suspect this issue is somewhat related to:&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-1895&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/CSHARP-1895&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;When I change the fetch function in the repro example to use the Sync API:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;async Task Fetch()&lt;br/&gt;
 {&lt;br/&gt;
&#160; await Task.Run(() =&amp;gt;&lt;br/&gt;
&#160; &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: {
&#160; &#160; collection.Find(filter).ToList();
&#160; &#160; collection.Aggregate&amp;lt;BsonDocument&amp;gt;(pipeline).ToList();
&#160; }&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;);&lt;br/&gt;
 }&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I get the following example after 30 seconds:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;System.TimeoutException: &apos;A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector&lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { AllowedLatencyRange = 00}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt; }. Client view of cluster state is { ClusterId : &quot;1&quot;, ConnectionMode : &quot;Automatic&quot;, Type : &quot;Unknown&quot;, State : &quot;Disconnected&quot;, Servers : [{ ServerId: &quot;&lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { ClusterId }&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;&quot;, EndPoint: &quot;Unspecified/localhost:27017&quot;, State: &quot;Disconnected&quot;, Type: &quot;Unknown&quot;, LastUpdateTimestamp: &quot;2019-08-14T09:37:22.4361867Z&quot; }] }.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Which is the same exception seen in 1895.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment>Windows 10 Version 1703&lt;br/&gt;
.NET Framework 4.6.1 &amp;amp; .NET Core 2.2&lt;br/&gt;
MongoDB 4.0.5 &amp;amp; 4.0.9</environment>
        <key id="894531">CSHARP-2689</key>
            <summary>Occasional extremely poor performance when connection pool is exhausted</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="robert@mongodb.com">Robert Stam</assignee>
                                    <reporter username="a.theodorsen@krohne.com">Anders Theodorsen</reporter>
                        <labels>
                    </labels>
                <created>Wed, 14 Aug 2019 09:58:02 +0000</created>
                <updated>Fri, 27 Oct 2023 13:24:55 +0000</updated>
                            <resolved>Mon, 18 Nov 2019 16:44:28 +0000</resolved>
                                    <version>2.8.1</version>
                                                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="2547098" author="rstam" created="Mon, 18 Nov 2019 16:44:14 +0000"  >&lt;p&gt;When writing an async program you need to be careful to be 100% async. If any of your code blocks it will end up affecting performance.&lt;/p&gt;

&lt;p&gt;&amp;gt;&#160;But is it really necessary to do two awaits to get the result for Aggregate asynchronously?&lt;/p&gt;

&lt;p&gt;Yes it is. The call to `AggregateAsync` returns a &lt;b&gt;cursor&lt;/b&gt;, and the call to `ToListAsync` reads everything up until the end of the cursor and returns the results as a List.&lt;/p&gt;

&lt;p&gt;If the query returns large amounts of data instead of calling `ToListAsync` to read all the data at once, you would want to work with the cursor directly to process a batch of results at a time.&lt;/p&gt;

&lt;p&gt;&amp;gt;As a side note, the documentation (&lt;a href=&quot;https://mongodb.github.io/mongo-csharp-driver/2.8/reference/driver/crud/reading/#aggregation&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mongodb.github.io/mongo-csharp-driver/2.8/reference/driver/crud/reading/#aggregation&lt;/a&gt;)&#160;also uses Aggregate().ToListAsync() as an async example which then can block in certain cases.&lt;/p&gt;

&lt;p&gt;There are two forms of the Aggregate methods:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The Aggregate(pipeline) and the AggregateAsync(pipeline) methods which return a cursor&lt;/li&gt;
	&lt;li&gt;The Aggregate() method which returns a fluent interface for building an aggregation pipeline&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The example you referred to is using the second form.&lt;/p&gt;</comment>
                            <comment id="2370441" author="a.theodorsen@krohne.com" created="Wed, 14 Aug 2019 12:50:24 +0000"  >&lt;p&gt;As a side note, the documentation (&lt;a href=&quot;https://mongodb.github.io/mongo-csharp-driver/2.8/reference/driver/crud/reading/#aggregation&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mongodb.github.io/mongo-csharp-driver/2.8/reference/driver/crud/reading/#aggregation&lt;/a&gt;)&#160;also uses Aggregate().ToListAsync() as an async example which then can block in certain cases.&lt;/p&gt;</comment>
                            <comment id="2370419" author="a.theodorsen@krohne.com" created="Wed, 14 Aug 2019 12:38:56 +0000"  >&lt;p&gt;D&apos;oh, it appears this problem is due to misuse on our side.&lt;/p&gt;

&lt;p&gt;I downloaded the source code for the Mongo C# driver and debugged. The problem occured in ExclusiveConnectionPool.cs:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;public&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; bool Wait(TimeSpan timeout, CancellationToken cancellationToken)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;return&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; _semaphore.Wait(timeout, cancellationToken);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;} &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/blockquote&gt;
&lt;p&gt;This causes the thread to block explaining the behaviour, so I traced the stack and it traces back to the .Aggregate() method, I didn&apos;t realize there is a AggregateAsync, and didn&apos;t think there was a need for it since .ToListAsync() is called after it at the end.&lt;/p&gt;

&lt;p&gt;So changing the &apos;Fetch&apos; function in the sample to the following solves the issue:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;async Task Fetch()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  await collection.Find(filter).ToListAsync();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  await (await collection.AggregateAsync&amp;lt;BsonDocument&amp;gt;(pipeline)).ToListAsync();&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&#160;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/blockquote&gt;
&lt;p&gt;But is it really necessary to do two awaits to get the result for Aggregate asynchronously?&lt;/p&gt;

&lt;p&gt;I also see the &apos;Find&apos; method has a &apos;FindAsync&apos; equivalent. When using the &apos;ToListAsync&apos; on them, is it still necessary/recommended to use the &apos;FindAsync&apos;? (And then double awaiting). As I haven&apos;t been able to see/reproduce the issue with only &apos;Find().ToListAsync()&apos; type queries.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="227132" name="MongoDbPerfIssue.zip" size="7552" author="a.theodorsen@krohne.com" created="Wed, 14 Aug 2019 09:22:36 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hv9lif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>