<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:14:49 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-47649] Possible race between session expiration and transaction creation</title>
                <link>https://jira.mongodb.org/browse/SERVER-47649</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We&apos;ve been observing odd transaction failures since ramping up an application that uses MongoDB 4.0 transactions.  We observe the following:&lt;br/&gt;
1) update, with lsid=X, txnNumber=1, startTransaction=true, autocommit=false&lt;br/&gt;
2) commitTransaction, with lsid=X, txnNumber=1, autocommit=false&lt;br/&gt;
3) wait for at least 30 minutes&lt;br/&gt;
4) update, with lsid=X, txnNumber=2, startTransaction=true, autocommit=false&lt;br/&gt;
5) 2nd update, with lsid=X, txnNumber=2, autocommit=false&lt;/p&gt;

&lt;p&gt;Despite the update in 4 succeeding, the update in 5 fails.  We&apos;ve added some debugging info to our mongod build and observe that when 5 fails, either _activeTxnNumber=1 and _txnState=3, or _activeTxnNumber=-1 and _txnState=0, which would indicate that mongod had seemingly lost the transaction state from 4.&lt;/p&gt;

&lt;p&gt;The actual operation/command doesn&apos;t seem matter, as we&apos;ve observed it with &quot;aggregate&quot; as well.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1320005">SERVER-47649</key>
            <summary>Possible race between session expiration and transaction creation</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="dmitry.agranat@mongodb.com">Dmitry Agranat</assignee>
                                    <reporter username="bartle">David Bartley</reporter>
                        <labels>
                    </labels>
                <created>Sat, 18 Apr 2020 02:26:37 +0000</created>
                <updated>Sun, 29 Oct 2023 22:09:17 +0000</updated>
                            <resolved>Sun, 31 May 2020 12:05:51 +0000</resolved>
                                                    <fixVersion>4.2.0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>18</watches>
                                                                                                                <comments>
                            <comment id="3147884" author="dmitry.agranat" created="Sun, 31 May 2020 12:05:28 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt;, I will go ahead and close this ticket given the reported issue is fixed in 4.2. If this is still an issue for you after upgrading to 4.2, please reopen this ticket and we&apos;ll have a look.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3068671" author="dmitry.agranat" created="Tue, 5 May 2020 07:09:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt; The race you described makes sense. In 4.0, SessionCatalog::_reapInMemorySessionsOlderThan &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r4.0.18/src/mongo/db/session_catalog.cpp#L362&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;acquired&lt;/a&gt; a mutex, then dropped and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r4.0.18/src/mongo/db/session_catalog.cpp#L375&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;re-acquired&lt;/a&gt; it, leaving a gap where a new transaction could be started but then still be treated as expired.&lt;/p&gt;

&lt;p&gt;This code was heavily refactored in 4.2 and we believe &lt;a href=&quot;https://github.com/mongodb/mongo/blame/321201823455e1c648b7c6bd4ae0a59d1b7115b8/src/mongo/db/session_catalog_mongod.cpp#L354&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this&lt;/a&gt; check in the new code prevents the race.&lt;/p&gt;</comment>
                            <comment id="3053390" author="dmitry.agranat" created="Thu, 23 Apr 2020 19:56:00 +0000"  >&lt;p&gt;Thanks &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt; for a detailed explanation of what might have happened. &lt;/p&gt;

&lt;p&gt;The reason I&apos;ve asked before about your driver and if you saw any client-side errors is because we&apos;ve seen (and fixed) at least one case with the same error which was related to client-side serialization error. Given the error was spotted only on the server-side, at first, it was not clear whether it&apos;s a driver, server, or another issue.&lt;/p&gt;

&lt;p&gt;Getting this information would help us to speed up this case progress together with the attempts to reproduce the reported issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3053060" author="jeff.yemin" created="Thu, 23 Apr 2020 17:39:54 +0000"  >&lt;p&gt;This does not look like SPEC-1195 to me.  SPEC-1195 is about client-side errors (e.g. a serialization error) that happens while in a transaction.&lt;/p&gt;</comment>
                            <comment id="3052510" author="jesse" created="Thu, 23 Apr 2020 12:49:10 +0000"  >&lt;p&gt;I&apos;ve moved teams since we wrote the Drivers Transactions Spec, so I&apos;m going to abstain from commenting.&lt;/p&gt;</comment>
                            <comment id="3052273" author="bartle" created="Thu, 23 Apr 2020 07:42:13 +0000"  >&lt;p&gt;Reading through the code in session.cpp and session_catalog.cpp, I have a potential explanation for what happend:&lt;br/&gt;
1) Existing session X is left idle for 30 mins&lt;br/&gt;
2) _reapInMemorySessionsOlderThan runs and observes that session X is expired, and adds it to possiblyExpiredLsids&lt;br/&gt;
3) X is used in a new transaction (i.e. update with startTransaction:true), and _activeTxnNumber is updated&lt;br/&gt;
4) _reapInMemorySessionsOlderThan continues running and calls _invalidateSession on X, which removes it from _txnTable&lt;br/&gt;
5) X is used in an additional transaction, but the session can&apos;t be found in _txnTable&lt;br/&gt;
6) refreshFromStorageIfNeeded is called to populate the in-memory session from disk, which loads details from the &quot;config.transactions&quot; collection, which would reflect state as of 1 (i.e. there would be no record of the new transaction from 3)&lt;/p&gt;

&lt;p&gt;In step 3 above, you might expect that this should also update the &quot;config.transactions&quot; collection (and hence 6 should have fetched state reflecting the new transaction), but my read of the code is that updates to &quot;config.transactions&quot; only happen in response to oplog writes (onWriteOpCompletedOnPrimary calls updateSessionEntry), and in 4.0 individual operations in a transaction don&apos;t generate an oplog write; instead, a single oplog entry is generated when the transaction is committed.&lt;/p&gt;

&lt;p&gt;Assuming this analysis is correct, I&apos;d expect the fix would be to have the &quot;Remove the session ids from the in-memory catalog&quot; loop verify that &quot;session-&amp;gt;lastCheckout &amp;lt; possiblyExpired&quot; is still true (in this case it wouldn&apos;t be, because the new session in 3) would have updated lastCheckout).&lt;/p&gt;</comment>
                            <comment id="3052265" author="bartle" created="Thu, 23 Apr 2020 07:11:32 +0000"  >&lt;p&gt;We&apos;re actually using a ~custom driver, though it should be compliant with the driver specs.  I have no reason to believe you couldn&apos;t reproduce this with a standard driver, though I don&apos;t know of a way to reliably reproduce this (other than perhaps trying to reuse a session in a transaction after ~exactly 30 min have past since the last transaction on the session).&lt;/p&gt;

&lt;p&gt;The error comes from &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r4.0.14/src/mongo/db/session.cpp#L567-L570&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r4.0.14/src/mongo/db/session.cpp#L567-L570&lt;/a&gt;, so the full error would be of the form &quot;Given transaction number 2 does not match any in-progress transactions.&quot;, with code NoSuchTransaction.&lt;/p&gt;

&lt;p&gt;There aren&apos;t any logs covering the time of the event, other than &quot;connection accepted&quot; and &quot;end connection&quot; logs.&lt;/p&gt;</comment>
                            <comment id="3046116" author="dmitry.agranat" created="Sun, 19 Apr 2020 09:52:28 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bartle&quot; class=&quot;user-hover&quot; rel=&quot;bartle&quot;&gt;bartle&lt;/a&gt;, thank you for the report.&lt;/p&gt;

&lt;p&gt;In order for us to investigate this issue, could you please provide:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;driver type and the exact driver version&lt;/li&gt;
	&lt;li&gt;compressed &lt;tt&gt;mongod&lt;/tt&gt; log covering the time of this event (or at least all the lines covering all the relevant isid and txnNumber)&lt;/li&gt;
	&lt;li&gt;full client-side error&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/4d0289a7-797c-485c-91cc-f0ed992d9731.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure uploader&lt;/a&gt; for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3045798" author="bartle" created="Sat, 18 Apr 2020 02:36:48 +0000"  >&lt;p&gt;Oh, the actual error is &quot;Given transaction number 2 does not match any in-progress transactions.&quot;.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 19 Apr 2020 09:52:28 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 36 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 36 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jesse@mongodb.com</customfieldvalue>
            <customfieldvalue>bartle</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>jeff.yemin@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxg527:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hx3ln3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxfrbj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>