<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:26:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31251] Increase backlog specified in listen()</title>
                <link>https://jira.mongodb.org/browse/SERVER-31251</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;A small value for the backlog parameter to listen() can result in very high latencies as seen by the client when an increase in application load or a decrease in db performance requires the client to create a large number of connections nearly simultaneously (i.e. a &quot;connections spike&quot;).&lt;/p&gt;

&lt;p&gt;When a connection spike occurs, mongod may not be able to accept() the connections as quickly as they are created. When the number of connections that have been created but not accepted by mongod exceeds the backlog, the SYN packets to establish the connections are ignored. The ignored clients then retry the SYN packets 1 second later, creating another smaller spike. If this spike also exceeds the backlog the result will be more ignored SYN packets, and another spike of retries 2 seconds later, and so on, with the clients doubling the backoff time on each attempt. Since the backoff is exponential, some unlucky connections can wait a very long time - tens of seconds - resulting in extreme operation latencies as seen by the client.&lt;/p&gt;

&lt;p&gt;This can be seen in the following two runs of the same client starts 5000 threads creating a connection spike of 5000 connections. The first run is with a backlog setting of 10000, while the second is with the default backlog of 128.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/166334/166334_backlog.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;At the default backlog setting on the right the network stats report a large number of &quot;listen drops&quot; and this results in the series of connection spikes of decreasing sizes and extreme latencies (as seen by the client) of ~64 s (&quot;slowest query durations&quot;) that we see here.&lt;/p&gt;

&lt;p&gt;With the larger backlog setting on the left there is no exponential backoff - connections are queued by the kernel for mongod to accept at the rate it can, eliminating the extreme latency outliers.&lt;/p&gt;

&lt;p&gt;3.6 introduces a parameter &quot;--listenBacklog&quot; to allow setting the backlog. However there are two problems with this:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;the default is SOMAXCONN (which is 128)&lt;/li&gt;
	&lt;li&gt;it isn&apos;t sufficient just to increase --listenBacklog, as the value is silently truncated at the value of the kernel parameter net.core.somaxconn, which by default is also 128, so the user must also increase this.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Rather than requiring the user to change two parameters, it would seem more straightforward for mongod to specify a large value in listen() by default, allowing the customer to control the backlog just by changing net.core.somaxconn.&lt;/p&gt;</description>
                <environment></environment>
        <key id="432576">SERVER-31251</key>
            <summary>Increase backlog specified in listen()</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="1" iconUrl="https://jira.mongodb.org/images/icons/statuses/open.png" description="">Open</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-servicearch">Backlog - Service Architecture</assignee>
                                    <reporter username="bruce.lucas@mongodb.com">Bruce Lucas</reporter>
                        <labels>
                            <label>features-not-sure-of</label>
                    </labels>
                <created>Mon, 25 Sep 2017 20:17:01 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:07 +0000</updated>
                                                            <fixVersion>features we&amp;#39;re not sure of</fixVersion>
                                    <component>Networking</component>
                                        <votes>4</votes>
                                    <watches>34</watches>
                                                                                                                <comments>
                            <comment id="2074362" author="bruce.lucas@10gen.com" created="Wed, 28 Nov 2018 18:53:06 +0000"  >&lt;p&gt;The observed behavior for status quo was different from that: the kernel simply drops the SYN packets when the backlog fills up, causing the client (at the network driver level) to go into an exponential backoff, the result of which is that some of the connections take a very long time to establish, but clients do not see ECONNREFUSED. An application may time out the connection attempt and retry before the connection is eventually established, but I think that just puts us at step 4 in your second scenario, so I&apos;m not sure there&apos;s a difference between the scenarios if establishing the connections is slow enough that apps time out on establishing connections.&#160; But in the case where&#160;mongod is able to accept the connections before the app begins to time out the connection attempts, then having a large backlog allows that to happen without the extreme latencies due to the exponential backoff.&lt;/p&gt;</comment>
                            <comment id="2074201" author="jason.carey" created="Wed, 28 Nov 2018 16:57:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=luke.prochazka&quot; class=&quot;user-hover&quot; rel=&quot;luke.prochazka&quot;&gt;luke.prochazka&lt;/a&gt; - there are more issues around connection establishment than just the listen backlog.  Changing that without addressing those other areas could easily make connection storms worse, rather than better.&lt;/p&gt;

&lt;p&gt;Consider two scenarios, one with a small list backlog, one with a large one -&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;StatusQuo&quot;&gt;&lt;/a&gt;Status Quo&lt;/h3&gt;

&lt;ol&gt;
	&lt;li&gt;Many client applications open connections to a host at once&lt;/li&gt;
	&lt;li&gt;mongod can&apos;t accept them fast enough&lt;/li&gt;
	&lt;li&gt;Listen backlog fills&lt;/li&gt;
	&lt;li&gt;Clients begin receiving ECONNREFUSED&lt;/li&gt;
	&lt;li&gt;Clients attempt new connections, those connections fail, eventually assume the host is down and potentially route traffic somewhere else&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;It&apos;s not great, and for primary required workloads the &quot;route somewhere else&quot; case may not be possible without a new election (which may trigger the whole problem again).&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;withadeeplistenbacklog&quot;&gt;&lt;/a&gt;with a deep listen backlog&lt;/h3&gt;

&lt;ol&gt;
	&lt;li&gt;Many client applications open connections to a host at once&lt;/li&gt;
	&lt;li&gt;mongod can&apos;t accept them fast enough&lt;/li&gt;
	&lt;li&gt;Listen backlog doesn&apos;t fill&lt;/li&gt;
	&lt;li&gt;Clients begin timing out sockets (connection timeout, socket timeout, overall time to connect/tls/auth, etc)&lt;/li&gt;
	&lt;li&gt;Server spins up threads to handle these sockets, does some work before the clients gives up (perhaps far too much work)&lt;/li&gt;
	&lt;li&gt;Clients attempt new connections, those connections fail, eventually assume the host is down and potentially route traffic somewhere else&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;The end result isn&apos;t that much different, but (for say a 64k backlog) we now have the potential to do 64k worth of wasted work, instead of 128.&lt;/p&gt;

&lt;p&gt;Things I&apos;d want to do before increasing the listen backlog:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Change the server to treat socket disconnection the same as killop for a running op (we currently finish any work we start for an operation, only noticing the clients gone away on send()) - this is in flight&lt;/li&gt;
	&lt;li&gt;Change drivers to treat connection failure as a rate limiting operation (this would be part of request backpressure)&lt;/li&gt;
	&lt;li&gt;Potentially re-write ingress client accept to make it multi-threaded (as it is, if we&apos;re hitting the current listen backlog, our single threaded acceptor is going to take multiple seconds to get around to your socket with a deeper one) (not planned)&lt;/li&gt;
	&lt;li&gt;Offer some kind of qos on which clients we prioritize actually running operations for (a world where allow more simultaneous connects will need hard queuing for running ops) (not planned)&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2065073" author="jason.carey" created="Fri, 16 Nov 2018 19:50:09 +0000"  >&lt;p&gt;I remain concerned that extending the size of the listen backlog is a poor bandaid for what we actually need (an ability to prevent clients from opening too many connections, and from interpreting connection failure as a signal to open many new connections).&lt;/p&gt;

&lt;p&gt;As such, I&apos;m lodging this in under the request backpressure scope, hoping that we can validate the use case it would solve via another channel&lt;/p&gt;</comment>
                            <comment id="2048150" author="acm" created="Wed, 31 Oct 2018 20:51:59 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mira.carey%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;mira.carey@mongodb.com&quot;&gt;mira.carey@mongodb.com&lt;/a&gt; - I&apos;m handing this ticket over to you, as it seems connected to the information that &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ben.caimano&quot; class=&quot;user-hover&quot; rel=&quot;ben.caimano&quot;&gt;ben.caimano&lt;/a&gt; was discussing about listener starvation.&lt;/p&gt;</comment>
                            <comment id="1827303" author="acm" created="Thu, 8 Mar 2018 15:39:05 +0000"  >&lt;p&gt;It is not safe to raise the default above SOMAXCONN. Instead, we plan in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31400&quot; title=&quot;Record Linux netstat metrics in ftdc&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31400&quot;&gt;&lt;del&gt;SERVER-31400&lt;/del&gt;&lt;/a&gt; to add improved network diagnostic data capture so that scenarios that would safely benefit from a listen backlog higher than SOMAXCONN can be more easily identified and addressed.&lt;/p&gt;</comment>
                            <comment id="1682186" author="bruce.lucas@10gen.com" created="Tue, 26 Sep 2017 16:56:37 +0000"  >&lt;p&gt;I tried various settings of net.ipv4.tcp_max_syn_backlog (500, the default 2048, 10000) in conjunction with a net.core.somaxconn of 10000, and did not find any difference in the behavior of the test application that creates a spike of 5000 connections. Since this is just the queue of connections awaiting completion of the 3-way handshake, I suspect the default setting of 2048 is ok.&lt;/p&gt;</comment>
                            <comment id="1682091" author="bruce.lucas@10gen.com" created="Tue, 26 Sep 2017 16:10:17 +0000"  >&lt;p&gt;I tested setting tcp_abort_on_overflow instead of increasing the backlog, but that results in &quot;connection reset&quot; errors being surfaced to the application, so I don&apos;t think that&apos;s a suitable substitute.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="440941">SERVER-31400</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="14736">SERVER-2554</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="166334" name="backlog.png" size="101540" author="bruce.lucas@mongodb.com" created="Mon, 25 Sep 2017 20:16:58 +0000"/>
                            <attachment id="166641" name="spike.png" size="40984" author="luke.prochazka@mongodb.com" created="Thu, 28 Sep 2017 11:22:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25132"><![CDATA[Service Arch]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000fMQTzQAO, 5002K00000msxWSQAY, 5002K00000nAuD8QAK, 5002K00000sblUgQAI, 5002K00000qcTpiQAE]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 25 Sep 2017 20:41:20 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 11 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<a href='https://jira.mongodb.org/browse/PM-1123'>PM-1123</a>, <s><a href='https://jira.mongodb.org/browse/PM-1211'>PM-1211</a></s>, <s><a href='https://jira.mongodb.org/browse/PM-1120'>PM-1120</a></s>, <s><a href='https://jira.mongodb.org/browse/PM-1124'>PM-1124</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_14262" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>End date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 29 Sep 2017 23:59:59 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1123</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 11 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-servicearch</customfieldvalue>
            <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>mira.carey@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htfo3b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htyzqv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2601">Service Arch 2018-11-05</customfieldvalue>
    <customfieldvalue id="2602">Service Arch 2018-11-19</customfieldvalue>
    <customfieldvalue id="2635">Service Arch 2018-12-03</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_14261" key="com.atlassian.jira.plugin.system.customfieldtypes:datepicker">
                        <customfieldname>Start date</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 25 Sep 2017 00:00:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htfa6n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>