<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:17:32 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-28232] Performance degradation after upgrade from 3.0.11 to 3.2.12</title>
                <link>https://jira.mongodb.org/browse/SERVER-28232</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Environment: CentoOS 6 (kernel 2.6.32), MongoDB community edition 3.0.11 and 3.2.12 , storage engine MMAP, config servers on SCCC&lt;/p&gt;

&lt;p&gt;We are experiencing performance degradation when moving from 3.0.11 to 3.2.12. Application throughput is getting reduced by 5-10 times in 3.2.12 compared to 3.0.11. In the past, we had attempted to upgrade from 3.0.11 to 3.2.8 but due to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26159&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-26159&lt;/a&gt; bug, we rollback to 3.0.11. In 3.2.8 application throughput was fine but since the mongos were randomly crashing due to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26159&quot; title=&quot;mongos crashes:  Invariant failure !op-&amp;gt;timedOut() src/mongo/executor/network_interface_asio.cpp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26159&quot;&gt;&lt;del&gt;SERVER-26159&lt;/del&gt;&lt;/a&gt; we rollback to 3.0.11. Bug &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26159&quot; title=&quot;mongos crashes:  Invariant failure !op-&amp;gt;timedOut() src/mongo/executor/network_interface_asio.cpp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26159&quot;&gt;&lt;del&gt;SERVER-26159&lt;/del&gt;&lt;/a&gt; fixed in 3.2.10 so we attempt to upgrade but we got our performance reduced so we rollback to 3.0.11 again. We opened a JIRA &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26654&quot; title=&quot;ExceededTimeLimit: Operation timed out after uprgade from 3.2.8 to 3.2.10&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26654&quot;&gt;&lt;del&gt;SERVER-26654&lt;/del&gt;&lt;/a&gt; about this issue (and several other people report almost the same issues) and according to Jira the issue was solved in 3.2.12. We attempt to upgrade to 3.2.12 but we got the same performance degradation as the 3.2.10 upgrade.&lt;/p&gt;

&lt;p&gt;The issue we are seeing in the logs after increasing the verbosity from 1 to 2 is the following:&lt;/p&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;I ASIO    [NetworkInterfaceASIO-TaskExecutorPool-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;] Failed to connect to (node) - ExceededTimeLimit: Operation timed out&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;D ASIO  [NetworkInterfaceASIO-TaskExecutorPool-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;] Failed to execute command: RemoteCommand &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;23628777&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; -- target:(node) db:admin cmd:{ &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;isMaster: &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; } reason: ExceededTimeLimit: Operation timed out&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;isMaster command is timeout &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; different &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;TaskExecutorPool&quot;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; all the time.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;


&lt;p&gt;Note: I am not changing the &quot;protocolVersion&quot; to 1 after the 3.0.11 to 3.2.12 upgrade as makes the rollback harder. &lt;/p&gt;

&lt;p&gt;We managed to reproduce the issue with sysbench-mongodb using 3.2.12 on a 10 nodes sharded cluster, not in the scale we getting it on our production system.&lt;/p&gt;

&lt;p&gt;To remedy the issue in testing we changed taskExecutorPoolSize value:&lt;/p&gt;

&lt;p&gt;Our mongos has 6 CPUs so I assume it creates 6 connection pools with defaults. Using a smaller value like &quot;taskExecutorPoolSize&quot;=2 reduces the timeouts so it seems the more connection pools I use the more timeouts I get during the benchmark. When I set &quot;taskExecutorPoolSize&quot;=1, which I believe set a single connection pool, I am not getting the above timeouts.&lt;/p&gt;

&lt;p&gt;We also modified the ShardingTaskExecutorPoolRefreshTimeoutMS from the default 20 seconds to 60 seconds which also eliminated the timeouts.&lt;/p&gt;

&lt;p&gt;We combined both on production but unfortunately, the timeouts didn&apos;t go away and we still noticed the same performance degradation.&lt;/p&gt;


&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;setParameter:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolRefreshTimeoutMS: &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;60000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; taskExecutorPoolSize: &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I want to believe that is not our workload that triggering the performance degradation as it operates fine on 3.0.11&lt;/p&gt;

&lt;p&gt;The purpose of the thread is to understand what has changed between 3.2.8 and 3.2.12 that might trigger the isMaster request to fail between mongod and mongos.&lt;/p&gt;

&lt;p&gt;Much appreciated If anyone has internals on the change or is facing the same problem and found a workaround.&lt;/p&gt;

&lt;p&gt;Thanks in advance,&lt;/p&gt;

&lt;p&gt;Jason&lt;/p&gt;</description>
                <environment></environment>
        <key id="362159">SERVER-28232</key>
            <summary>Performance degradation after upgrade from 3.0.11 to 3.2.12</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="jason_or">Jason Terpko</reporter>
                        <labels>
                    </labels>
                <created>Tue, 7 Mar 2017 16:58:56 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:05 +0000</updated>
                            <resolved>Mon, 27 Nov 2017 21:25:48 +0000</resolved>
                                    <version>3.2.12</version>
                                                    <component>Performance</component>
                                        <votes>3</votes>
                                    <watches>19</watches>
                                                                                                                <comments>
                            <comment id="1735498" author="thomas.schubert" created="Mon, 27 Nov 2017 21:25:48 +0000"  >&lt;p&gt;Since 3.2.15 has been out for some time, and there hasn&apos;t been any additional activity on this ticket, I&apos;m going to resolve it. If this is still an issue after upgrading, please let us know and we&apos;ll continue to investigate.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1735496" author="thomas.schubert" created="Mon, 27 Nov 2017 21:22:17 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=dharshanr%40scalegrid.net&quot; class=&quot;user-hover&quot; rel=&quot;dharshanr@scalegrid.net&quot;&gt;dharshanr@scalegrid.net&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Please feel free to review our &lt;a href=&quot;https://docs.mongodb.com/manual/reference/parameters/#general-parameters&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;documentation&lt;/a&gt; on these parameters. If you have suggestions to improve our documentation, would you please open a &lt;a href=&quot;https://jira.mongodb.org/projects/DOCS/issues/DOCS-11057?filter=allopenissues&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;DOCS ticket&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1723694" author="dharshanr@scalegrid.net" created="Mon, 13 Nov 2017 01:32:29 +0000"  >&lt;p&gt;Hi folks,&lt;/p&gt;

&lt;p&gt;Is it possible to get the details of the connPoolMaxShardedConnsPerHost documented? A  few more questions on Sam&apos;s comments above&lt;br/&gt;
1)Is this limit per target host (connected from mongos) or is it for the mongos server in total?&lt;br/&gt;
2) Is this connPoolMaxConnsPerHost parameter required for connPoolMaxShardedConnsPerHost? If this parameter is only for mongos can we skip connPoolMaxConnsPerHost right?&lt;br/&gt;
3) Is there only one pool per server or does it depend on the number of cores&lt;/p&gt;</comment>
                            <comment id="1605593" author="jason.carey" created="Fri, 23 Jun 2017 20:15:17 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=antogiann&quot; class=&quot;user-hover&quot; rel=&quot;antogiann&quot;&gt;antogiann&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;That sounds good, I&apos;m happy to leave this open until 3.2.15 comes out and revisit then.&lt;/p&gt;</comment>
                            <comment id="1604136" author="antogiann" created="Thu, 22 Jun 2017 10:35:10 +0000"  >&lt;p&gt;A brief update:  &lt;/p&gt;

&lt;p&gt;We are currently deploying 3.2.14 with the following settings to production clusters&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolRefreshTimeoutMS: 600000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We have started deployment using a busy ascending order and so far the workaround is solid.  &lt;/p&gt;

&lt;p&gt;We would like to keep the ticket open until 3.2.15 is out. Currently, there is a cost during a mongos restart as it has to initialize the connections in advance (fixed cost: ShardingTaskExecutorPoolMinSize*Shards* connection pools). it&apos;s not a huge issue but we would like to investigate the impact of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-29237&quot; title=&quot;Add maxConnecting in asio connpool&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-29237&quot;&gt;&lt;del&gt;SERVER-29237&lt;/del&gt;&lt;/a&gt; to this behavior. &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ant&lt;/p&gt;</comment>
                            <comment id="1573461" author="samantha.ritter@10gen.com" created="Wed, 17 May 2017 15:48:47 +0000"  >&lt;p&gt;Hi Ant,&lt;/p&gt;

&lt;p&gt;Yes, you are seeing log messages from two different families of pools here.  For historical reasons, we use two separate kinds of connection pools on the write path and on the read path.&lt;/p&gt;

&lt;p&gt;The read path uses the NetworkInterfaceASIO pools.  All of the &lt;tt&gt;ShardingTaskExecutorPool*&lt;/tt&gt; parameters that you&apos;ve been working with affect NetworkInterfaceASIO pools only.&lt;/p&gt;

&lt;p&gt;On the write path, we use connection pools called DBClient pools.  These have a default maximum capacity of 200.  The fact that you&apos;re seeing so many log messages about connections in this pool being dropped means you are churning connections, because you consistently need more than 200 at any given time.  So, we open more connections as-needed to meet the demands on the pool, but we only ever keep 200 of them around for re-use.  I would suggest that you set this pool size to something higher, and see if that helps with both your performance issues and helps decrease the churn and log messages.  You can do this with the following:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;--setParameter connPoolMaxConnsPerHost=300&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;and&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;--setParameter connPoolMaxShardedConnsPerHost=300&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;When examining logging, the component part of the message should tell you which part of the system it&apos;s coming from.  The component is the part of the log line in square brackets.  The NetworkInterfaceASIO pools (the read path) log with component &lt;tt&gt;NetworkInterfaceASIO&lt;/tt&gt;, and the DBClient-based (write) pools log with various other components, based on what the pool is being used for.&lt;/p&gt;

&lt;p&gt;Let me know if this makes sense, I know there are a lot of moving parts here.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Sam&lt;/p&gt;</comment>
                            <comment id="1573175" author="antogiann" created="Wed, 17 May 2017 12:46:29 +0000"  >&lt;p&gt;I was testing with 3.2.13 version and the following settings&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolRefreshTimeoutMS: 600000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;and I received tons of Informational messages (27436 in 1 hour) of the type:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Ending idle connection to host &amp;lt;&lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;hostname&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&amp;gt;:&amp;lt;&lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;port&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&amp;gt;(with timeout of 0 seconds) because the pool meets constraints; 246 connections to that host remain open&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The number of connections on the message varies but its constantly over 200. &lt;/p&gt;

&lt;p&gt;Is there a maxPoolSize=200 limit enforced on each pool? or this is a different pool?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ant&lt;/p&gt;</comment>
                            <comment id="1570693" author="jonhyman" created="Sun, 14 May 2017 15:00:54 +0000"  >&lt;p&gt;Hi Jason, Wayne has tried setting various configuration values in our case to no avail. The last configuration we tried was this:&lt;/p&gt;

&lt;p&gt;setParameter:&lt;br/&gt;
  ShardingTaskExecutorPoolRefreshTimeoutMS: 60000&lt;br/&gt;
  ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;br/&gt;
  ShardingTaskExecutorPoolMinSize: 125&lt;/p&gt;

&lt;p&gt;We did not set a max pool size, I am fearful of limiting connections. We have thousands of processes which connect to our mongoS, in total we run tens of thousands of application server threads which may talk to the database and I am fearful of limiting connections to something as small as 200*num_mongos*num_cores_per_mongos.&lt;/p&gt;

&lt;p&gt;In thinking about this ticket and our problems, are you continuing to do more investigation based on these settings and what works to make changes to how the router works, or is your current approach that we should continue to tweak these settings as needed? The problem as I see with the latter that when the ExceededTimeLimit issue occurs, we have downtime of some kind due to read/write failures to the mongoS. It&apos;s a really bad situation when that happens. &lt;/p&gt;

&lt;p&gt;The instances that we&apos;re running 3.2.12 on right now have lower load by more than an order of magnitude from our busiest mongo instances, so I am extremely concerned that even if we get these settings right on one instance as Antonios has &amp;#8211; we now have a 3.2.12 instance that we haven&apos;t seen the problem on, but it&apos;s about 4-10x smaller/less busy than the one Wayne is reporting problems on, which is at least 10x smaller/less busy than our busiest instances &amp;#8211; that we&apos;re going to be relying on fragile configuration that may not support more application server growth or throughput.&lt;/p&gt;</comment>
                            <comment id="1570327" author="antogiann" created="Sat, 13 May 2017 00:56:44 +0000"  >&lt;p&gt;Hi Jason,&lt;/p&gt;

&lt;p&gt;At the moment we run only one 3.2.12 sharded cluster in production with:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;We made the change on May 08 and so far we haven&apos;t seen any &quot;ExceededTimeLimit: Operation timed out&quot; exception. The change seems to be on the right direction but compared to our other clusters, this one is considered as quiet from traffic perspective, so we can&apos;t assume that is an acceptable workaround before we test it against a busier cluster, which is an action item for the near future.&lt;/p&gt;

&lt;p&gt;Beyond production deployment, I tested various configuration to our test environment that we are able to reproduce the exception using mongobench.&lt;/p&gt;

&lt;p&gt;Before each test I restart the mongos and application servers and the duration of each the test is 1 hour.&lt;/p&gt;

&lt;p&gt;On the first round of tests I was increasing the traffic on the cluster gradually by adding more application servers on the test:&lt;/p&gt;

&lt;p&gt;Configuration01: We didn&apos;t hit the exception &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/p&gt;

&lt;p&gt;Configuration02: We hit the exception  on random times &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 1200000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration03: We hit the exception  on random times &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolMinSize: 100 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration04: We hit the exception  on random times &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 1200000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration05: We hit the exception  on random times &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 300000000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration05: We hit the exception on random times &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 1200000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;taskExecutorPoolSize: 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;On the second round of tests I was starting all application servers at once - in order to simulate a spike:&lt;/p&gt;

&lt;p&gt;Configuration06: We hit the exception&apos; when the application servers started for around 30 seconds and then didn&apos;t appear again.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 3600000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration07: We hit the exception&apos; when the application servers started for around 1-2 minutes and then didn&apos;t appear again.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 300000000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 200 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration08: We hit the exception&apos; when the application servers started for around 10 seconds and then didn&apos;t appear again.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: 300000000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Configuration09: We didn&apos;t hit the exception &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolHostTimeoutMS: &apos;300000000&apos; &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ShardingTaskExecutorPoolRefreshTimeoutMS: 600000 &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;For configurations06-08 I assume we got the exception in the beggining as the connection pool wasn&apos;t fully initialized yet.  For test Configuration07 (which was the ugliest) I also received exceptions of the type &quot;ExceededTimeLimit: Couldn&apos;t get a connection within the time limit&quot; which also lead me to the conclusion that mongos needs time to prepare the pools/connections. For  Configuration09 I believe the ShardingTaskExecutorPoolRefreshTimeoutMS absorve the errors but (as expected) it didn&apos;t help the mongos to prepare the pools faster (the application start reported traffic almost the same time as Configuration08). The only observation I can&apos;t explain is why Configuration06 produced more errors than Configuration08. My assumption was that they will produce the same. &lt;/p&gt;

&lt;p&gt;The next step was to test configurations06-08 without restarting the mongos (not a cold restart) . During these tests we didn&apos;t hit the exception. I assume because of the fact that the pools/connections were already there. Additionally during these tests the application reported traffic faster than the ones I restarted the mongos (the application servers were still restarted). &lt;/p&gt;

&lt;p&gt;My next of set (ongoing) is adding various values for &quot;taskExecutorPoolSize&quot; for Configuration09. When I finalize the optimal configuration which I think it will be Configuration09 I am also going to test it against global/partial step-downs.&lt;/p&gt;

&lt;p&gt;My conclusion so far is that the connection spikes seems to causing the abnormal behavior, so maintaining open connections is the right path. &lt;/p&gt;

&lt;p&gt;I am not sure if in this phase of testing I need to add ShardingTaskExecutorPoolMaxSize on the equation. I might give it a try in later stages. Theoretically the value of 200 will limit the concurrent operations to 200*num_mongos*num_connection_pools, correct?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ant&lt;/p&gt;</comment>
                            <comment id="1570244" author="jason.carey" created="Fri, 12 May 2017 21:24:01 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jonhyman&quot; class=&quot;user-hover&quot; rel=&quot;jonhyman&quot;&gt;jonhyman&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wegerer&quot; class=&quot;user-hover&quot; rel=&quot;wegerer&quot;&gt;wegerer&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Responding here, to your comments on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28629&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28629&quot;&gt;&lt;del&gt;SERVER-28629&lt;/del&gt;&lt;/a&gt;.  I believe this ticket includes the logs related to your problem, so I&apos;d like to keep the conversation here if possible.&lt;/p&gt;

&lt;p&gt;I&apos;ve made an attempt to correlate some events in the logs you uploaded to our support site and I have a story for how 3.2.8 might have helped you, as well as why it may not be a full solution.&lt;/p&gt;

&lt;p&gt;I filtered my view down to one of the uploaded app servers, and then to egress connections to one particular shard.  I&apos;m omitting the names here as they aren&apos;t otherwise available on this ticket and I&apos;m not sure if you want those names to be public.  For what counts as an egress connection, it&apos;s anything logged by the task executor with &quot;Connecting to&quot;, &quot;Successfully connected to&quot; or &quot;Failed to connect to&quot;&lt;/p&gt;

&lt;p&gt;Given that view, I see the queue of &quot;connecting&quot; connections rise from 0 to a peak of 46 from 7:09:06 - 7:09:30.  59 total connections are opened during that time.&lt;/p&gt;

&lt;p&gt;Then, from 7:12:41 - 7:14:04, 206 total connections are opened, peaking with 159 in the &quot;connecting&quot; state&lt;/p&gt;

&lt;p&gt;Finally, from 7:19:58 - 7:21:12, 160 connections are opened, before the first &quot;ExceededTimeLimit&quot; appears, where previously 261 connections were in the &quot;connecting&quot; state.&lt;/p&gt;

&lt;p&gt;That&apos;s a behavior pattern that might indicate bursts of traffic, with connection dumping in between (in which case setting min pool size would help by preventing that dumping).&lt;/p&gt;

&lt;p&gt;It might also indicate bursts of new traffic, as those log lines would present in the same way if we simply added more load.  In that latter case, you might do better to set max pool size, as that would throttle the opening of new connections and prevent the connections spikes that you&apos;re clearly experiencing.&lt;/p&gt;

&lt;p&gt;If it&apos;s a mixture of both, and you don&apos;t have too many long running operations, you might be best served by a combination of both approaches:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;setParameter=ShardingTaskExecutorPoolHostTimeoutMS=3600000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;setParameter=ShardingTaskExecutorPoolMinSize=100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;setParameter=ShardingTaskExecutorPoolMaxSize=200&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Such a configuration would only drop connections from a pool after an hour without traffic and would steadily maintain in the range of 100-200 connections to any given host.  It seems that your shards are capable of absorbing around 100 new connections in a burst, but drown out and enter timeout cycles at around 200.&lt;/p&gt;

&lt;p&gt;In the same vein as my comment to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=antogiann&quot; class=&quot;user-hover&quot; rel=&quot;antogiann&quot;&gt;antogiann&lt;/a&gt;, I&apos;d be interested in seeing what your logs look like with the various mongos options enabled.&lt;/p&gt;</comment>
                            <comment id="1569782" author="jason.carey" created="Fri, 12 May 2017 15:55:44 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=antogiann&quot; class=&quot;user-hover&quot; rel=&quot;antogiann&quot;&gt;antogiann&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Now that a few days have passed, how has the workaround described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28629&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28629&quot;&gt;&lt;del&gt;SERVER-28629&lt;/del&gt;&lt;/a&gt; worked out for you?&lt;/p&gt;

&lt;p&gt;I&apos;d be interested in knowing if things have stayed stable for you, or what sort of behavior you&apos;ve been seeing if not.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Jason&lt;/p&gt;</comment>
                            <comment id="1566681" author="antogiann" created="Mon, 8 May 2017 12:35:20 +0000"  >&lt;p&gt;Hi, &lt;/p&gt;

&lt;p&gt;We deployed in production (version 3.2.12) one of the workarounds described on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28629&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28629&quot;&gt;&lt;del&gt;SERVER-28629&lt;/del&gt;&lt;/a&gt;:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolHostTimeoutMS: 300000000&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ShardingTaskExecutorPoolMinSize: 100&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The ShardingTaskExecutorPoolMinSize came from the fact that the shards were maintain around 2000 connections prior to the change. We are running with 4 mongos and each mongos has 6 CPUs (so 6 connection pools) so I am expecting to have at least 4*6*100=2400 connections which is close to what the cluster needs.&lt;/p&gt;

&lt;p&gt;There is no good reason why we picked ShardingTaskExecutorPoolHostTimeoutMS= 300000000. It was the only value that during the benchmarks in our test envairoment didn&apos;t produced the exception but as the test continues we might be able to determine a lower value for it. &lt;/p&gt;

&lt;p&gt;So far I haven&apos;t noticed any &quot;Failed to connect to - ExceededTimeLimit: Operation timed out&quot; on the logs but its only 4 hours since we made the change. I will have some feedback after 24-48 hours and when the ShardingTaskExecutorPoolHostTimeoutMS gets exhausted. &lt;/p&gt;

&lt;p&gt;Ant&lt;/p&gt;</comment>
                            <comment id="1562200" author="antogiann" created="Tue, 2 May 2017 11:13:03 +0000"  >&lt;p&gt;Hi, I noticed that 3.2.13 released last week. I am going to try to reproduce the issue against 3.2.13 this week but I was wondering if there are any improvements on logging that might help us reveal the source of the error? &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ant&lt;/p&gt;</comment>
                            <comment id="1547250" author="acm" created="Wed, 12 Apr 2017 16:18:51 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=antogiann&quot; class=&quot;user-hover&quot; rel=&quot;antogiann&quot;&gt;antogiann&lt;/a&gt; - If you are running 2.12 then I believe we can rule out the glibc bug as a cause. Thanks for letting us know: it is helpful to us to eliminate that as a potential cause for the performance issue you are experiencing.&lt;/p&gt;</comment>
                            <comment id="1546412" author="antogiann" created="Tue, 11 Apr 2017 21:07:37 +0000"  >&lt;p&gt;Hi Thomas&lt;/p&gt;

&lt;p&gt;Its 2.12.  I am aware of &lt;a href=&quot;https://bugzilla.kernel.org/show_bug.cgi?id=99671&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugzilla.kernel.org/show_bug.cgi?id=99671&lt;/a&gt; so I had strace logging 3.2.12 mongos calls during the benchmark but I didn&apos;t catch any abnormal behavior. &lt;br/&gt;
Do you believe is related to glibc version? (If I recall from previous tickets one of the users reported the issue was using Windows.) Is there any diagnostic that will help us understand if glibc is the issue or not?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ant&lt;/p&gt;</comment>
                            <comment id="1546176" author="thomas.schubert" created="Tue, 11 Apr 2017 17:46:05 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jonhyman&quot; class=&quot;user-hover&quot; rel=&quot;jonhyman&quot;&gt;jonhyman&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We&apos;re continuing to investigate this issue. To help us rule out some theories, would you please provide the version of glibc that you are using by executing the following command?&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ldd --version&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas &lt;/p&gt;</comment>
                            <comment id="1545981" author="jonhyman" created="Tue, 11 Apr 2017 15:10:15 +0000"  >&lt;p&gt;Hi @Thomas Schubert, do you have an update on this ticket? I also wanted to point out &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28629&quot; title=&quot;router blocks and throws ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28629&quot;&gt;&lt;del&gt;SERVER-28629&lt;/del&gt;&lt;/a&gt; which may be related to this.&lt;/p&gt;</comment>
                            <comment id="1526434" author="thomas.schubert" created="Fri, 17 Mar 2017 04:34:57 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason_or&quot; class=&quot;user-hover&quot; rel=&quot;jason_or&quot;&gt;jason_or&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We&apos;re continuing to examine the data you&apos;ve provided. The heavy load I identified previously (a spike in commands/s) does not appear to be correlated to timeouts recorded in the logs. We&apos;ll update this ticket as our investigation continues.&lt;/p&gt;

&lt;p&gt;Thank you for your patience,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1523743" author="jason_or" created="Tue, 14 Mar 2017 16:56:26 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;Were you able to review the additional information and provided logs?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Jason&lt;/p&gt;</comment>
                            <comment id="1519615" author="jason_or" created="Thu, 9 Mar 2017 04:02:42 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;During the upgrade process the application was paused at times allowing for actions like mongos restarts, step downs, or if the application queue backlogged. The application was paused at the start of that window to perform a rolling mongos restart, at this time we increased taskExecutorPoolSize from 1 to 2 as test prior to rolling back.&lt;/p&gt;

&lt;p&gt;The application was un-paused at various times during that ~1 hour log sample. Pausing and un-pausing of the application during this process is normal and the clusters handle this just fine pre-3.2.10. A very similar workload (i.e. throughput) ran fine on different cluster running 3.2.8 but we encountered &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26159&quot; title=&quot;mongos crashes:  Invariant failure !op-&amp;gt;timedOut() src/mongo/executor/network_interface_asio.cpp&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26159&quot;&gt;&lt;del&gt;SERVER-26159&lt;/del&gt;&lt;/a&gt; and had to rollback. The application was again un-paused after rolling back to 3.0.11 with no issues reported.&lt;/p&gt;

&lt;p&gt;Because the diagnostics files are binary we can&apos;t read them, what metric in the file showed significant load? This should be due to new connections from the rolling mongos restart and un-pausing of the application. I am collecting the mongos log files now to be uploaded to the portal. Minus authentication and query log lines.&lt;/p&gt;

&lt;p&gt;Note, my team member Antonios may also be commenting on this Jira issue based on availability.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Jason&lt;/p&gt;</comment>
                            <comment id="1519081" author="thomas.schubert" created="Wed, 8 Mar 2017 17:10:05 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason_or&quot; class=&quot;user-hover&quot; rel=&quot;jason_or&quot;&gt;jason_or&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I see that between 2017-03-02T15:44:35.027 UTC and 2017-03-02T15:46:21.476Z UTC, your servers have significantly more load. If not, can you attempt to correlate correlate this time to the time outs in the mongos? Are you able to upload the mongos logs to the secure portal as well?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1518361" author="jason_or" created="Tue, 7 Mar 2017 22:02:42 +0000"  >&lt;p&gt;Thomas,&lt;/p&gt;

&lt;p&gt;Thank you for confirming the contents of diagnostic.data, I have uploaded metrics from the primaries while running version 3.2.12. According to the log these were the primaries after the first step down but were secondaries at times during the upgrade process. Looking at the log on one of the mongos servers these members logged the most occurrences of &quot;ExceededTimeLimit: Operation timed out&quot;.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Jason&lt;/p&gt;</comment>
                            <comment id="1518288" author="thomas.schubert" created="Tue, 7 Mar 2017 21:06:08 +0000"  >&lt;p&gt;Yes, that&apos;s correct diagnostic.data does not contain any user data. Please feel free to review the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.2.12/src/mongo/db/ftdc/ftdc_mongod.cpp#L314&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;source code&lt;/a&gt; to confirm.&lt;/p&gt;

&lt;p&gt;I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/89e4cff0-71be-4875-83a3-c79d6e8198eb.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; for you to use. Files uploaded to this portal are only visible to MongoDB employees investigating this issue and are routinely deleted.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1518281" author="jason_or" created="Tue, 7 Mar 2017 20:57:00 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;Thank you for assisting, can you confirm the contents of diagnostic.data only include the following data?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;serverStatus: db.serverStatus({tcmalloc: &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;replSetGetStatus: rs.status()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;collStats &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; local.oplog.rs: db.getSiblingDB(&lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&apos;local&apos;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;).oplog.rs.stats()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;getCmdLineOpts: db.adminCommand({getCmdLineOpts: &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;buildInfo: db.adminCommand({buildInfo: &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;hostInfo: db.adminCommand({hostInfo: &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Can you also please provide a secure method for uploading these files? One that is not accessible by the public.&lt;/p&gt;

&lt;p&gt;Jason&lt;/p&gt;</comment>
                            <comment id="1518117" author="thomas.schubert" created="Tue, 7 Mar 2017 19:05:18 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason_or&quot; class=&quot;user-hover&quot; rel=&quot;jason_or&quot;&gt;jason_or&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please upload an archive of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory from a primary node in the sharded cluster and identify when this issue was affecting it?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>25.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 7 Mar 2017 18:43:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 11 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 11 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>antogiann</customfieldvalue>
            <customfieldvalue>dharshanr@scalegrid.net</customfieldvalue>
            <customfieldvalue>jason_or</customfieldvalue>
            <customfieldvalue>jonhyman</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>mira.carey@mongodb.com</customfieldvalue>
            <customfieldvalue>samantha.ritter@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht3srb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hswbsv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4exz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>