<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:09:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-25663] Odd connection timeouts and rejections when replicaset secondary is lagged</title>
                <link>https://jira.mongodb.org/browse/SERVER-25663</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We were performing a rather massive set of updates. During these updates, one of the secondaries began to show SEND_ERROR messages in the logs, and also began to lag behind the primary.  The CSHARP client was not catching the errors, causing everything to be inoperable from a client standpoint.&lt;/p&gt;

&lt;p&gt;The only values that show up in the logfiles are:&lt;br/&gt;
2016-08-17T10:53:57.994-0500 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn886174&amp;#93;&lt;/span&gt; SocketException handling request, closing client connection: 9001 socket exception &lt;span class=&quot;error&quot;&gt;&amp;#91;SEND_ERROR&amp;#93;&lt;/span&gt; server &lt;span class=&quot;error&quot;&gt;&amp;#91;webserver:55640&amp;#93;&lt;/span&gt;&lt;/p&gt;


&lt;p&gt;&amp;#8211; I will also file a report on this with the CSHARP crew, as it appears to be unhandled coming back to the client.&lt;/p&gt;</description>
                <environment></environment>
        <key id="309727">SERVER-25663</key>
            <summary>Odd connection timeouts and rejections when replicaset secondary is lagged</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="sallgeud">Chad Kreimendahl</reporter>
                        <labels>
                    </labels>
                <created>Wed, 17 Aug 2016 17:27:54 +0000</created>
                <updated>Wed, 1 Feb 2017 00:47:10 +0000</updated>
                            <resolved>Wed, 1 Feb 2017 00:47:10 +0000</resolved>
                                    <version>3.2.8</version>
                                                    <component>Networking</component>
                    <component>Stability</component>
                                        <votes>1</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1489634" author="thomas.schubert" created="Wed, 1 Feb 2017 00:46:58 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for confirming you haven&apos;t encountered &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25070&quot; title=&quot;Significant delays when large collections are responsible for full WiredTiger cache&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25070&quot;&gt;&lt;del&gt;SERVER-25070&lt;/del&gt;&lt;/a&gt; since upgrading to MongoDB 3.2.10. My understanding is that this issue has been resolved as well, so I&apos;m updating the ticket. Please let me know if that isn&apos;t the case.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1369264" author="thomas.schubert" created="Fri, 26 Aug 2016 17:00:15 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the information. I&apos;ve created a secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/836ad61f-1ce7-4dd2-9ca7-2542989a44f3.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you to use.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1369203" author="sallgeud" created="Fri, 26 Aug 2016 16:23:17 +0000"  >&lt;p&gt;1. mongodump -o /backups/repl1&lt;br/&gt;
2. 3.2.8&lt;br/&gt;
3. Thousands. ~100&lt;br/&gt;
4. yes (a secondary)&lt;br/&gt;
5. at first near 0. By the end, it has large spikes. Will get exact numbers when I can test after-hours&lt;br/&gt;
6. about 60% (nearly all by mongod)&lt;br/&gt;
7. secure upload?&lt;/p&gt;</comment>
                            <comment id="1361309" author="thomas.schubert" created="Thu, 18 Aug 2016 19:23:41 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the additional information. I have a few questions so we can continue to investigate what is happening when the mongodump is executed.&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Would you please provide the command lines for mongodump that you have executed?&lt;/li&gt;
	&lt;li&gt;Which version of mongodump are you using?&lt;/li&gt;
	&lt;li&gt;How many collections are in your system? How many databases?&lt;/li&gt;
	&lt;li&gt;Is the mongodump being executed on the same machines as the mongod instance?&lt;/li&gt;
	&lt;li&gt;How much memory is mongodump using?&lt;/li&gt;
	&lt;li&gt;How much memory is typically used when mongodump isn&apos;t running?&lt;/li&gt;
	&lt;li&gt;Please upload the &lt;tt&gt;diagnostic.data&lt;/tt&gt; of the affected mongod.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thank you again for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1361219" author="sallgeud" created="Thu, 18 Aug 2016 18:15:00 +0000"  >&lt;p&gt;It appears that the slowness started approximately 20 minutes into a &quot;mongodump&quot; backup being performed on the secondary in question. When these mongodump processes run, they eat up every available ounce of memory eventually, sometimes forcing mongod to use swap (vm.swapiness=1 because swap is bad but OOM is worse). Based on observations, their is either some form of memory leak in mongodump, or some highly unnecessary usage of memory. This appears to remain true whether you use any flags (gzip, j, etc) or not.&lt;/p&gt;

&lt;p&gt;In this low memory situation, with some data in mongod swapping, we get enormously long queries. Finds that typically take 2 - 10ms begin to take between 5 and 100 seconds. It was in this scenario where the problem began. The SEND_ERROR we were seeing is likely the client side nuking the connection because it took too long.&lt;/p&gt;

&lt;p&gt;Specifically we had an empty find that was returning 25 records from a collection take 100+ seconds. That seems outside the realm of swap alone being the problem.  &quot;iostat&quot; on the system at the time showed nearly no disk activity, other than mongodump writes (which appear to happen in bulk)&lt;/p&gt;</comment>
                            <comment id="1359814" author="sallgeud" created="Wed, 17 Aug 2016 17:33:48 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-1748&quot; title=&quot;Not catching certain error scenarios from replicaset members&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CSHARP-1748&quot;&gt;&lt;del&gt;CSHARP-1748&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="301191">SERVER-25070</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="309730">CSHARP-1748</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 18 Aug 2016 19:23:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 2 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 2 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjypr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsote7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;The scenarios used to make this happen are not easily reproducible. Our setup is a 3 member replicaset in which one of them becomes angry enough to start closing connections.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlfev:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>