<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:21:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49998] MongoS task executor stalls intermittently while establishing connections</title>
                <link>https://jira.mongodb.org/browse/SERVER-49998</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Description&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Periodically our MongoS (version 3.6.16) task executors experience intermittent increased latency while establishing new connections to MongoD. The latency occurs between the&#160;&lt;tt&gt;ASIOConnection::setup&lt;/tt&gt;&apos;s call to&#160;&lt;tt&gt;strand()-&amp;gt;dispatch()&lt;/tt&gt;&#160;and the dispatch lambda being invoked. This can delay the call to&#160;&lt;tt&gt;NetworkInterfaceASIO::_connect&lt;/tt&gt;&#160;for up to 20 seconds.&lt;/p&gt;

&lt;p&gt;The latency seems to max out at 20 seconds, indicating it may involve the&#160;&lt;tt&gt;options.refreshTimeout&lt;/tt&gt;&#160;which is also used as a timeout during connection establishment.&lt;/p&gt;

&lt;p&gt;There is no external cause or correlation we can find. Stalls occur randomly / intermittent, and seem to occur during otherwise normal conditions.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Runtime Environment&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;We run MongoS in Kubernetes 1.15.12, Docker 18.6.1, kernel 5.4.15-1.el7.elrepo.x86_64.&lt;/p&gt;

&lt;p&gt;Our Docker image for&#160;&lt;tt&gt;/usr/bin/mongos&lt;/tt&gt;&#160;is built from&#160;&lt;tt&gt;&quot;deb &lt;a href=&quot;http://repo.mongodb.org/apt/ubuntu&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://repo.mongodb.org/apt/ubuntu&lt;/a&gt; xenial/mongodb-org-unstable/3.6 multiverse&quot;&lt;/tt&gt;&#160;official MongoDB package&#160;&lt;tt&gt;mongodb-org-mongos:3.6.16&lt;/tt&gt;&#160;and the binary is unmodified.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Sequence of Events&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;The task executor stall and related side effects occur as follows:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Queries arrive at our MongoS instance and call&#160;&lt;tt&gt;ClusterFind::runQuery&lt;/tt&gt;&#160;which selects the next task executor and calls&#160;&lt;tt&gt;mongo::establishCursors&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;ConnectionPool::SpecficPool::getConnection(hostAndPort)&lt;/tt&gt;&#160;is called.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;ConnectionPool::SpecificPool::spawnConnections&lt;/tt&gt;&#160;is called, which sees the available connection pool is too small and constructs a new connection, then calls&#160;&lt;tt&gt;ASIOConnection::setup&lt;/tt&gt;&#160;to establish the connection and add it to the ready pool.&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;ASIOConnection::setup&lt;/tt&gt;&#160;dispatches a callback to the ASIO strand with&#160;&lt;tt&gt;_impl-&amp;gt;strand().dispatch()&lt;/tt&gt;&#160;and a lambda callback&#160;{{&lt;span class=&quot;error&quot;&gt;&amp;#91;this, timeout, cb&amp;#93;&lt;/span&gt; { ... }}}&lt;/li&gt;
	&lt;li&gt;Something occurs which delays invoking the dispatch lambda callback.
	&lt;ul&gt;
		&lt;li&gt;We see this latency take up to 20 seconds maximum.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;This&#160;&lt;tt&gt;ASIOConnection::setup&lt;/tt&gt;&#160;process repeats for each&#160;&lt;tt&gt;getConnection(host:port)&lt;/tt&gt;&#160;pair on the stalled task executor.
	&lt;ul&gt;
		&lt;li&gt;Maximum number of simultaneous &quot;connecting&quot; connections defaults to 2. If the&#160;&lt;tt&gt;getConnection&lt;/tt&gt;&#160;were called on 6 distinct&#160;&lt;tt&gt;host:port&lt;/tt&gt;&#160;endpoints we can see the number of &quot;connecting&quot; connections spike to 12. This increases the &quot;refreshing&quot; connection count in the pool stats.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Queries queue up at the stalled task executor waiting for an available connection.&lt;/li&gt;
	&lt;li&gt;Applications connected using the Java driver SRV-based cluster select among servers randomly. The stalled server&apos;s connection pool in the client becomes exhausted as all connections are waiting on stalled queries.&lt;/li&gt;
	&lt;li&gt;Applications gradually begin opening additional connections to the server with the stalled task executor, attempting to increase throughput.
	&lt;ul&gt;
		&lt;li&gt;Some fraction of these new queries will also become queued at the stalled task executor.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;The affected MongoS instance sees a gradual increase in client connections.&lt;/li&gt;
	&lt;li&gt;Other task executors in the affected MongoS continue processing queries and maintain normal latencies.
	&lt;ul&gt;
		&lt;li&gt;Only the stalled executor shows increased query latencies, since queries are not being executed and thus don&apos;t reach a MongoD instance for many seconds.&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Finally the dispatch lambda callbacks are invoked, which calls&#160;&lt;tt&gt;ASIOConnection::setTimeout&lt;/tt&gt;&#160;and then initiates the connection process by calling&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp#L251&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;_global-&amp;gt;_impl-&amp;gt;_connect(_impl.get());&lt;/tt&gt;&lt;/a&gt;&#160;at line 251.&lt;/li&gt;
	&lt;li&gt;No connection-related errors are observed in the logs.&lt;/li&gt;
	&lt;li&gt;Latency between&#160;&lt;tt&gt;_connect()&lt;/tt&gt;&#160;and&#160;&lt;tt&gt;_setupCallback&lt;/tt&gt;&#160;lambda calls appear normal (comparable to the steady state connect times of a few milliseconds). The latency only increases between&#160;&lt;tt&gt;strand()-&amp;gt;dispatch()&lt;/tt&gt;&#160;and&#160;&lt;tt&gt;_connect()&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;Application clients observe increased latency for some random subset of queries executed on the stalled instance during the stall event.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&lt;b&gt;Observations&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Using&#160;&lt;a href=&quot;https://github.com/iovisor/bpftrace&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bpftrace&lt;/a&gt;&#160;I&apos;ve measured the latency between the following points inside&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;connection_pool_asio.cpp&lt;/tt&gt;&lt;/a&gt; - all line numbers are for the r3.6.16 version of the source:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;At the start of the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp#L207&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;ASIOConnection::setup&lt;/tt&gt;&lt;/a&gt;&#160;method invoked at line 207.&lt;/li&gt;
	&lt;li&gt;Immediately before the call to&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp#L242&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;{{setTimeout(timeout, [this, access, generation] { ... }}}&lt;/a&gt;&#160;at line 242. This indicates our lambda has been invoked by&#160;&lt;tt&gt;strand().dispatch(..)&lt;/tt&gt;&#160;and we are about to call&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp#L251&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;_global-&amp;gt;_impl-&amp;gt;_connect()&lt;/tt&gt;&lt;/a&gt;. Latency is delta from (1).&lt;/li&gt;
	&lt;li&gt;At the start of execution of the&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.16/src/mongo/executor/connection_pool_asio.cpp#L209&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;_setupCallback&lt;/tt&gt;&lt;/a&gt;&#160;lambda defined at line 209. This indicates the completion of the&#160;&lt;tt&gt;_connect&lt;/tt&gt;&#160;operation. Latency is delta from (1).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;In steady state we see the latency hovering around a few milliseconds. At certain times the task executor stalls for up to 20 seconds between points (1) and (2). We believe this may involve the refresh timeout (default 20 seconds) but have not yet been able to trace the root cause.&lt;/p&gt;

&lt;p&gt;Attached is a&#160;&lt;a href=&quot;https://github.com/sqsp-scratchpad/phensley-cms-perf/blob/master/2020-07-28/setup-latency.bt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;bpftrace script&lt;/a&gt;&#160;and a&#160;&lt;a href=&quot;https://github.com/sqsp-scratchpad/phensley-cms-perf/blob/master/setup-latency.log&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;log showing the latencies&lt;/a&gt;&#160;during a single query stall event.&lt;/p&gt;

&lt;p&gt;For example, at 22:44:54.411 the first call to&#160;&lt;tt&gt;1. ASIOConnection::setup()&lt;/tt&gt;&#160;is made for connection handle 93874978094592. This takes 16.821 seconds to reach&#160;&lt;tt&gt;2. call setTimeout()&lt;/tt&gt;&#160;at 22:45:11.232 and&#160;&lt;tt&gt;3. in _setupCallback&lt;/tt&gt;&#160;about 7ms after that.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Graphs&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;**I&apos;ve attached graphs showing the client connection spike associated with the task executor stall (setup-latency-client-connections.png) as well as a 3-hour and 48-hour graph showing the frequency over time.&lt;/p&gt;

&lt;p&gt;Stalls occur with varying frequency depending on the volume of request traffic going to a particular MongoS instance. The probability of a stall increases with increasing traffic &amp;#8211; rate of stalls is lower overnight and higher at peak traffic mid-day. Higher traffic tiers experience more frequent spikes than lower traffic tiers.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1424028">SERVER-49998</key>
            <summary>MongoS task executor stalls intermittently while establishing connections</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13204">Community Answered</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="phensley@squarespace.com">Patrick Hensley</reporter>
                        <labels>
                    </labels>
                <created>Wed, 29 Jul 2020 15:27:56 +0000</created>
                <updated>Fri, 27 Oct 2023 15:56:31 +0000</updated>
                            <resolved>Mon, 5 Oct 2020 07:15:47 +0000</resolved>
                                    <version>3.6.16</version>
                                                    <component>Sharding</component>
                                        <votes>1</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="3402686" author="dmitry.agranat" created="Mon, 21 Sep 2020 07:57:46 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=phensley%40squarespace.com&quot; class=&quot;user-hover&quot; rel=&quot;phensley@squarespace.com&quot;&gt;phensley@squarespace.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Our internal investigation concluded that the reported incident (i.e., intermittent stalls) is potentially due to using ServiceExecutorAdaptive, which is an experimental feature, not enabled by default, and depreciated in new versions. If the same behavior is observed after switching back to the default service executor, we&apos;ll be happy to investigate the underlying cause.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3329801" author="dmitry.agranat" created="Mon, 10 Aug 2020 09:55:41 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=phensley%40squarespace.com&quot; class=&quot;user-hover&quot; rel=&quot;phensley@squarespace.com&quot;&gt;phensley@squarespace.com&lt;/a&gt;, I am passing this to one of our teams to analyze. &lt;/p&gt;</comment>
                            <comment id="3324091" author="phensley@squarespace.com" created="Thu, 6 Aug 2020 08:48:37 +0000"  >&lt;p&gt;The &lt;tt&gt;mongos-pod-mnzmb-client-connections.png&lt;/tt&gt; overview graph is in NY time UTC-4, so 13:35 refers to the first event starting around 17:35:00 UTC.&lt;/p&gt;

&lt;p&gt;The &lt;tt&gt;setup-latency-mongos-pod-mnzmb.log&lt;/tt&gt; file is in UTC, and the first dispatch/connect stall occurs at&#160;17:35:43.034 for connection handle&#160;94440919596544, which is dispatched 9.7 seconds later at 17:35:53.735:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;17&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;35&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;43.034&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;48013&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; conn27847         &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;. ASIOConnection::setup() &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;94440919596544&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;17&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;35&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;53.735&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;22462&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; Network.ool-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;. call setTimeout()       &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;94440919596544&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;9700&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;17&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;35&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;53.743&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;22462&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; Network.ool-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;. in _setupCallback       &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;94440919596544&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;9709&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;The &lt;tt&gt;mongos-server.log&lt;/tt&gt; is also in UTC, and at&#160;17:35:43.008 client connections gradually increase from 646 up to a peak of 996 at 17:35:58.362. By&#160;17:37:50 all excess connections have gone idle and been closed by the application connection pools.&lt;/p&gt;</comment>
                            <comment id="3324069" author="dmitry.agranat" created="Thu, 6 Aug 2020 07:54:11 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=phensley%40squarespace.com&quot; class=&quot;user-hover&quot; rel=&quot;phensley@squarespace.com&quot;&gt;phensley@squarespace.com&lt;/a&gt;, could you please provide times in UTC we should be looking at? From the notes I am not sure what &lt;tt&gt;13:35&lt;/tt&gt; refers to, is it UTC -4?&lt;/p&gt;</comment>
                            <comment id="3323272" author="phensley@squarespace.com" created="Wed, 5 Aug 2020 18:27:59 +0000"  >&lt;p&gt;Hi Dima, I&apos;ve collected diagnostics files and server logs for a new capture, along with the other related files, and have uploaded them via the support uploader tool.&lt;/p&gt;</comment>
                            <comment id="3317684" author="phensley@squarespace.com" created="Mon, 3 Aug 2020 15:13:13 +0000"  >&lt;p&gt;Hi Dima, since we run mongoS under Kubernetes/Docker we log to stdout. Since we don&apos;t set the &lt;tt&gt;logpath&lt;/tt&gt; option on startup, that apparently fails to create the&#160;&lt;tt&gt;diagnostic.data&lt;/tt&gt; directory.&lt;/p&gt;

&lt;p&gt;I&apos;ll look into getting that enabled on a single pod in production and capturing another tracing session, and uploading it via the support uploader.&lt;/p&gt;</comment>
                            <comment id="3317028" author="dmitry.agranat" created="Sun, 2 Aug 2020 09:37:39 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=phensley%40squarespace.com&quot; class=&quot;user-hover&quot; rel=&quot;phensley@squarespace.com&quot;&gt;phensley@squarespace.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for a detailed report. &lt;/p&gt;

&lt;p&gt;For the mongoS in question and covering the time of the event, would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/b1de5f52-eece-4060-930c-6efd53010dc5.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="273233" name="Screen Shot 2020-08-10 at 12.54.08.png" size="416465" author="dmitry.agranat@mongodb.com" created="Mon, 10 Aug 2020 09:58:24 +0000"/>
                            <attachment id="271818" name="setup-latency-client-connections-3-hours.png" size="139003" author="phensley@squarespace.com" created="Wed, 29 Jul 2020 15:22:52 +0000"/>
                            <attachment id="271817" name="setup-latency-client-connections-48-hours.png" size="172743" author="phensley@squarespace.com" created="Wed, 29 Jul 2020 15:22:55 +0000"/>
                            <attachment id="271819" name="setup-latency-client-connections.png" size="105610" author="phensley@squarespace.com" created="Wed, 29 Jul 2020 15:22:36 +0000"/>
                            <attachment id="271821" name="setup-latency.bt" size="1578" author="phensley@squarespace.com" created="Wed, 29 Jul 2020 15:38:28 +0000"/>
                            <attachment id="271822" name="setup-latency.log" size="93013" author="phensley@squarespace.com" created="Wed, 29 Jul 2020 15:38:33 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 29 Jul 2020 18:58:10 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 20 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 20 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>phensley@squarespace.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxxauf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxk3m7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;We have not yet been able to observe or reproduce the issue in a non-production environment.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxwx3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>