<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:51:40 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-61147] Ensure safe deletion of TenantMigrationRecipientAccessBlocker for Shard Merge &amp; MTM.</title>
                <link>https://jira.mongodb.org/browse/SERVER-61147</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;del&gt;See design doc for details. &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-61141&quot; title=&quot;Create TenantMigrationRecipientAccessBlocker for each tenant&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-61141&quot;&gt;&lt;del&gt;SERVER-61141&lt;/del&gt;&lt;/a&gt; should&apos;ve accomplished what we need. Check that the multitenant migrations tests already cover all the&#160;TenantMigrationRecipientAccessBlocker cases we need to test for Merge.&lt;/del&gt;&lt;/p&gt;

&lt;p&gt;EDIT:&lt;/p&gt;

&lt;p&gt;In-memory RTAB is used to reject user donor snapshot reads earlier than &lt;a href=&quot;https://github.com/10gen/mongo/blob/4db2628c4897f7fc2fa2e7e2555008f7cbc8752f/src/mongo/db/repl/tenant_migration_state_machine.idl#L212&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rejectReadsBeforeTimestamp&lt;/a&gt;&#160; after shard merge commit because the first version of shard merge doesn&apos;t preserve/copy donor history. In-memory RTAB is deleted once the recipient state document is deleted. Currently, TTL delete the state document after shard merge completion with GC delay of &lt;a href=&quot;https://github.com/10gen/mongo/blob/64c4305a9b1f4149c08c3eb91d31e749d24d2e56/src/mongo/db/repl/repl_server_parameters.idl#L430&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;tenantMigrationGarbageCollectionDelayMS&lt;/a&gt; (default is 15 mins). This is risky as there is no guarantee that R&#8217;s oldest timestamp &amp;gt;= rejectReadsBeforeTimestamp after GC delay. So, after merge commit, we can have readers trying to do donor data reads at snapshot earlier than rejectReadsBeforeTimestamp after GC delay. And, there will be nothing to prevent those readers from reading inconsistent data, violating snapshot read guarantees.&lt;/p&gt;

&lt;p&gt;(The same problem exists in MTM protocol as well).&lt;/p&gt;</description>
                <environment></environment>
        <key id="1913715">SERVER-61147</key>
            <summary>Ensure safe deletion of TenantMigrationRecipientAccessBlocker for Shard Merge &amp; MTM.</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="1" iconUrl="https://jira.mongodb.org/images/icons/statuses/open.png" description="">Open</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-servicearch">Backlog - Service Architecture</assignee>
                                    <reporter username="jesse@mongodb.com">A. Jesse Jiryu Davis</reporter>
                        <labels>
                    </labels>
                <created>Sat, 30 Oct 2021 12:23:51 +0000</created>
                <updated>Mon, 16 Oct 2023 21:04:45 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5047844" author="suganthi.mani" created="Mon, 12 Dec 2022 15:43:58 +0000"  >&lt;p&gt;Andy, Esha and myself agreed with good-enough solution i.e, GC window to be &amp;gt; &amp;gt; minSnapshotHistoryWindow and is a very small effort solution.  So, we doesn&apos;t need to bring this into product&apos;s radar. &lt;/p&gt;</comment>
                            <comment id="5004697" author="suganthi.mani" created="Wed, 23 Nov 2022 19:03:49 +0000"  >&lt;p&gt;As per offline discussion with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi@mongodb.com&quot;&gt;esha.maharishi@mongodb.com&lt;/a&gt;, since this is a bug both in MTM and shard merge, moving this ticket out of Shard Merge project and putting into serverless backlog&lt;/p&gt;</comment>
                            <comment id="4456134" author="suganthi.mani" created="Mon, 4 Apr 2022 20:38:09 +0000"  >&lt;blockquote&gt;&lt;ol&gt;
	&lt;li&gt;We decided not to proceed with the hack &lt;tt&gt;GC delay &amp;gt; snapshot window&lt;/tt&gt;, instead directly implement the cleaner solution to address the problem which will garbage collect (delete) the state document only when the node&apos;s current oldest timestamp is greater than&#160;&lt;a href=&quot;https://github.com/10gen/mongo/blob/57b670b4df8b5ad731306a811500b46c9e71a0e9/src/mongo/db/repl/tenant_migration_state_machine.idl#L214&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;rejectReadsBeforeTimestamp&lt;/a&gt;&#160;and not involve the TTL to garbage collect the document via expireAt field.&lt;/li&gt;
	&lt;li&gt;Couple of ways to implement the cleaner solution that came out of first round of brainstorming with Esha.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Approach#1 Use the recipient state machinery to do async polling of node&apos;s oldest timestamp and deletion of state document.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;con: We can still have a slight race where the donor snapshot reads can be retried on stale R secondary whose oldest timestamp is not yet &amp;gt; rejectReadsBeforeTimestamp but the state doc got deleted. To be noted, donor primary communicates the majority commit point through heartbeat / oplog channel . Only on the majority commit point advancement, stable ts &amp;amp; oldest time moves forward.&lt;/li&gt;
	&lt;li&gt;pro: changes are just in recipient state machine code.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Approach#2: Register a hook to&#160;&lt;tt&gt;WiredTigerKVEngine]()::setOldestTimestampFromStable]&lt;/tt&gt;&#160;to delete the state document when current oldest timestamp is greater than rejectReadsBeforeTimestamp.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;con: Complexity in writing the new hook , scanning &amp;amp; disk writes as part of heartbeat or oplog application code path can lead to unintended consequences(not sure though)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Approach#3: Use a separate reaper service to monitor the oldest timestamp and deletion of state document.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;con: Use of a new thread pool for the reaper service.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;&lt;/blockquote&gt;</comment>
                            <comment id="4456132" author="suganthi.mani" created="Mon, 4 Apr 2022 20:36:56 +0000"  >&lt;p&gt;Making GC delay &amp;gt; snapshot window can still be racy and lead to incorrect data response for snapshot reads (see &lt;a href=&quot;https://github.com/10gen/mongo/pull/4199#issuecomment-1084831490&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; for more discussion on it)&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;My understanding is that TTL monitor expires/deletes the state document based on wall clock time. But minSnapshotHistory window doesn&apos;t move based on wall clock time. Let me try to explain what I am thinking with an example (sorry if it&apos;s too complicated)&lt;/p&gt;

&lt;p&gt;Let&apos;s say GC delay time is set to 6 mins &amp;gt; Snapshot history window (= 5 mins).&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;R writes the state document &apos;expireAt&apos; field to 8.06 A.M (wall clock time + GC delay time) at a timestamp with wall clock time equivalent to 8.00 A.M&lt;/li&gt;
	&lt;li&gt;Say the majority replication lag is 3 mins. This would cause the above written state document to get majority committed delayed by 3 mins . This in turn implies that only at wall clock time 8.03 A.M, the stable timestamp on R will be at a timestamp with wall clock time equivalent to 8.00 A.M.
	&lt;ol&gt;
		&lt;li&gt;This indicates that only&#160;&lt;b&gt;after&lt;/b&gt;&#160;wall clock time 8.08 A.M, any reads that tries to do a snapshot at a timestamp with wall clock time equivalent to 8.00 A.M, would get &quot;SnasphotTooOld&quot; error. ((&lt;a href=&quot;https://github.com/10gen/mongo/blob/57b670b4df8b5ad731306a811500b46c9e71a0e9/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp#L2257-L2259&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;see&lt;/a&gt;&#160;here how they calculate the history window).&lt;/li&gt;
	&lt;/ol&gt;
	&lt;/li&gt;
	&lt;li&gt;Say, our current wall clock time is 8.07 A.M, so now the TTL monitor deletes the state document written in step # 2. This would result in tenant access blocker to get removed.&lt;/li&gt;
	&lt;li&gt;Now, Say, our current wall clock time is 8.08 A.M, the client tries to read donor data at a snapshot at a timestamp with wall clock time equivalent to 7.59 A.M. This would not return &quot;SnapshotTooOld&quot; error as we don&apos;t have access blocker and that&apos;s a bug.&lt;/li&gt;
&lt;/ol&gt;
&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="4437416" author="jesse" created="Sun, 27 Mar 2022 19:40:33 +0000"  >&lt;p&gt;The existing tests cover new Shard Merge functionality. However, we&apos;re planning to enable snapshot reads in Serverless and we must ensure access blockers remain for minSnapshotHistoryWindowInSeconds. If minSnapshotHistoryWindowInSeconds &amp;lt; tenantMigrationGarbageCollectionDelayMS:&lt;/p&gt;

&lt;p&gt;1. Client starts snapshot session on donor D.&lt;br/&gt;
2. D replies to client&apos;s first query with read timestamp T.&lt;br/&gt;
3. Client stores T, and continues querying D at T.&lt;br/&gt;
4. D donates all its data to recipient R and the migration commits.&lt;br/&gt;
5. Client attempts to query R at T, R correctly replies &quot;SnapshotTooOld&quot; since T predates the migration.&lt;br/&gt;
6. After tenantMigrationGarbageCollectionDelayMS, R deletes its access blockers.&lt;br/&gt;
7. If the client is somehow still trying to query R at T, R will attempt to read tenant data at T. R had no data for the tenant at that time, so it will reply with empty results. This is incorrect, R should reply with &quot;SnapshotTooOld&quot;.&lt;/p&gt;

&lt;p&gt;The default tenantMigrationGarbageCollectionDelayMS is 15 minutes, the default minSnapshotHistoryWindowInSeconds is 5 minutes, so this bug won&apos;t usually be possible. But let&apos;s either enforce this relationship, or set the GC delay to max(tenant GC delay, snapshot window) to be sure.&lt;/p&gt;</comment>
                            <comment id="4291918" author="jesse" created="Thu, 13 Jan 2022 20:59:08 +0000"  >&lt;p&gt;Ensure that the TenantMigrationRecipientAccessBlocker rejects such reads with &quot;SnapshotTooOld&quot; if the readConcern&apos;s &quot;atClusterTime&quot; predates the merge. Ensure access blockers remain for at least minSnapshotHistoryWindowInSeconds.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1913709">SERVER-61141</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2071823">SERVER-67372</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1867449">SERVER-59794</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2016366">SERVER-65236</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25132"><![CDATA[Service Arch]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 28 Mar 2022 14:26:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 8 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-61141'>SERVER-61141</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-67372'>SERVER-67372</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>dbeng-pm-bot</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 8 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jesse@mongodb.com</customfieldvalue>
            <customfieldvalue>backlog-server-servicearch</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i08cpr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i06nln:r</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i07yv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>