<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:44:28 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-36923] Scoped connections not being returned from Logical Sessions Cache machinery</title>
                <link>https://jira.mongodb.org/browse/SERVER-36923</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h3&gt;&lt;a name=&quot;Investigation&quot;&gt;&lt;/a&gt;Investigation&lt;/h3&gt;
&lt;p&gt;Scoped connection errors exist in a &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_windows_64_2k8_logical_session_cache_refresh_jscore_passthrough_100_patch_529b7e260862c8748212f1158764b4a93f967eda_5b85a7822a60ed26040c2b99_18_08_28_19_50_51##%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%2522529b7e260862c8748212f1158764b4a93f967eda%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;custom logical session refresh jsCore passthrough suite&lt;/a&gt;. This suite sets the logical session refresh time to 100ms. &lt;/p&gt;

&lt;p&gt;At the end of each job, secondaries cannot create a connection to the primary because &quot;only one usage of each socket address (protocol/network address/port) is normally permitted.&quot; &lt;/p&gt;

&lt;p&gt;Throughout the job&apos;s lifetime, we see instances of &quot;scoped connection&quot; ... &quot;not being returned to the pool&quot; from all nodes.&lt;/p&gt;

&lt;p&gt;This is the ordering of events I was able to put together from the highlighted lines on the &lt;a href=&quot;https://logkeeper.mongodb.org/lobster/build/d5e84f1404a3ef0db3e414c86d7fe496/all#bookmarks=0%2C234%2C509%2C528%2C566%2C21519%2C21525%2C21526%2C21527%2C21528%2C21529%2C21530%2C21538%2C21539%2C22746%2C22764%2C22832%2C27332&amp;amp;f=00not%20being%20&amp;amp;f=00Only%20one%20usage%20of%20each%20socket%20address&amp;amp;f=00ReplicaSetFixture%3Ajob3%3Aprimary&amp;amp;f=10ReplicaSetFixture%3Ajob3%3Aprimary.%2ALogicalSessionCache&amp;amp;f=10conn3004&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Lobster log for one of the jobs&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;1. The LogicalSessionCache Refresh thread on the primary starts a replica set monitor on itself (is this normal?)&lt;br/&gt;
2. The primary can&#8217;t find the primary (itself) in context of the replica set&lt;br/&gt;
3. The primary eventually finds the primary in context of the replica set&lt;br/&gt;
3. The primary and both secondaries creates scoped connections to the primary that are being terminated without being in a fail state (&#8230;&quot;scoped connection&quot;&#8230; &quot;not being returned to the pool&quot;)&lt;br/&gt;
4. Client requests restart of the primary (I&apos;m not sure if this actually happens, but why else would the CONTROL startup lines be repeated so late in the test?)&lt;br/&gt;
5. The LogicalSessionCache Refresh and Reap threads on secondaries now get connpool errors attempting to connect to the primary&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;Problem&quot;&gt;&lt;/a&gt;Problem&lt;/h3&gt;
&lt;p&gt;We are letting ScopedDBConnections go out of scope before calling done() &amp;#8211; the call to done() places the connection back on the pool.&lt;/p&gt;

&lt;h3&gt;&lt;a name=&quot;Proposedfix&quot;&gt;&lt;/a&gt;Proposed fix&lt;/h3&gt;
&lt;p&gt;Call done() on the connection before the connection goes out of scope.&lt;/p&gt;</description>
                <environment></environment>
        <key id="597325">SERVER-36923</key>
            <summary>Scoped connections not being returned from Logical Sessions Cache machinery</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="blake.oler@mongodb.com">Blake Oler</assignee>
                                    <reporter username="blake.oler@mongodb.com">Blake Oler</reporter>
                        <labels>
                    </labels>
                <created>Wed, 29 Aug 2018 18:09:00 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:15 +0000</updated>
                            <resolved>Wed, 5 Sep 2018 14:46:15 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1991002" author="misha.tyulenev" created="Fri, 31 Aug 2018 16:55:35 +0000"  >&lt;p&gt;Good point &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blake.oler&quot; class=&quot;user-hover&quot; rel=&quot;blake.oler&quot;&gt;blake.oler&lt;/a&gt;! while it does not remove the race - i.e. its still possible to get this error by opening the connection to the primary from the reaper and the refreshSessions threads &lt;/p&gt;</comment>
                            <comment id="1990518" author="blake.oler" created="Fri, 31 Aug 2018 00:25:11 +0000"  >&lt;p&gt;It turns out the issue was unrelated to races between the reap/refresh threads. Whenever we send a command to an RS primary from SessionsCollection RS, we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a3d17a55ca68ba37eb59620e04258f61f133b21f/src/mongo/db/sessions_collection_rs.cpp#L123&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;drop the reference to the ScopedDBConnection&lt;/a&gt; without calling done() on it. Adding the call to done() removes all connection pool errors and the end-of-job socket errors.&lt;/p&gt;

&lt;p&gt;Whether we should still approach serialization of the reap and refresh threads &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=misha.tyulenev&quot; class=&quot;user-hover&quot; rel=&quot;misha.tyulenev&quot;&gt;misha.tyulenev&lt;/a&gt; is an open question. Let&apos;s discuss this tomorrow.&lt;/p&gt;</comment>
                            <comment id="1989990" author="jason.carey" created="Thu, 30 Aug 2018 15:28:11 +0000"  >&lt;p&gt;I&apos;m happy with adding a mutex to the sessions collection (i.e. synchronizing in that type, rather than adding logic to the cache)&lt;/p&gt;</comment>
                            <comment id="1989329" author="misha.tyulenev" created="Wed, 29 Aug 2018 20:57:52 +0000"  >&lt;p&gt;Looks to me that we have a race between Reap and Refresh threads. Need to synchronize the access and it should be gone. I have updated the description with the proposed fix section.&lt;/p&gt;</comment>
                            <comment id="1989233" author="blake.oler" created="Wed, 29 Aug 2018 19:34:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=misha.tyulenev&quot; class=&quot;user-hover&quot; rel=&quot;misha.tyulenev&quot;&gt;misha.tyulenev&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Discovered some more odd behavior in another symptomatic suite. Check out &lt;a href=&quot;https://logkeeper.mongodb.org/lobster/build/0c756362b390ccb7f0a28ce9b5946485/all#bookmarks=0%2C503%2C507%2C518%2C519%2C520%2C538%2C759%2C27947&amp;amp;f=00CONTROL&amp;amp;f=01network&amp;amp;f=01REPL_HB&amp;amp;f=00not%20being%20returned%20to%20the%20pool&amp;amp;f=10LogicalSessionCacheRefresh&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this lobster&lt;/a&gt;:&lt;/p&gt;

&lt;p&gt;For all three hosts, &lt;em&gt;two&lt;/em&gt; connections to the primary are made almost simultaneously. Each connection thinks it&apos;s the first connection and reports only one connection active. This seems to be why one connection isn&apos;t being killed properly. This happens once in two places in the Refresh thread (on secondary1), and twice (primary, secondary0) with the Refresh/Reap threads opening a connection simultaneously.&lt;/p&gt;

&lt;p&gt;I think it has to do with the call to setupSessionsCollection, because the double connect happens soon after the processes stop complaining that &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;I have an Evergreen build running &lt;a href=&quot;https://evergreen.mongodb.com/version/5b86f1a40305b93fe8f068ee&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; to verify this is the case.&lt;/p&gt;

&lt;p&gt;I haven&apos;t achieved any progress with understanding why the primary was restarted in the job analyzed in the ticket description. However, not all jobs analyzed had a node restart, so I&apos;m going to focus on the double connect behavior for now&lt;/p&gt;</comment>
                            <comment id="1989141" author="misha.tyulenev" created="Wed, 29 Aug 2018 18:21:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blake.oler&quot; class=&quot;user-hover&quot; rel=&quot;blake.oler&quot;&gt;blake.oler&lt;/a&gt; sessions_collection_rs.cpp need to find out the topoly of RS so it uses RemoteCommandTargeterRS which takes the connection string and either finds the RSM in the existing global pool or create new. Its normal if it happens once per connection string per process (not thread)&lt;br/&gt;
However in the log the line &quot;Starting new replica set monitor for rs/localhost:20750&quot;&lt;br/&gt;
looks suboptimal to me because its just one host for a 3 node RS. This is also the reason it cant find the primary. Later RSM discovers the correct string in the isMaster response. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="516137">SERVER-34120</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 29 Aug 2018 18:21:21 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 23 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1237</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 23 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>blake.oler@mongodb.com</customfieldvalue>
            <customfieldvalue>mira.carey@mongodb.com</customfieldvalue>
            <customfieldvalue>misha.tyulenev@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu6rhz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htxa9z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2481">Sharding 2018-09-10</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu6drb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>