<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:44:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-58409] Startup RecordId initialization is flawed with durable history and reconstructing prepared transactions</title>
                <link>https://jira.mongodb.org/browse/SERVER-58409</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;RecordIds by MDB are generated with an auto-incrementing integer. Initialization on a restart, or after rollback opens a cursor (with data at the stable timestamp) and does one reverse step to get the largest id in use for a document.&lt;/p&gt;

&lt;p&gt;Now consider the following sequence with durable history:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Set OldestTimestamp 1&lt;/li&gt;
	&lt;li&gt;Insert RecordId(1) -&amp;gt; A at TimeStamp(10)&lt;/li&gt;
	&lt;li&gt;Insert RID(2) -&amp;gt; B at TS(20)&lt;/li&gt;
	&lt;li&gt;Delete RID(2) (B) at TS(30)&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If we were to restart and initialize the next record id, we&apos;d start issuing new documents RID(2). Normally this is fine. Any new replicated user writes must be generated with a timestamp larger than 30, so the update chain for RID(2) will remain valid.&lt;/p&gt;

&lt;p&gt;However, when reconstructing prepared transactions, the prepare timestamp (and thus any following commit timestamp, but not the durable timestamp) may be arbitrarily old.&lt;/p&gt;

&lt;p&gt;In this example, after initializing the next RID to 2, if we were to reconstruct a prepared transaction from TS(10) that performs an insert on this collection, we&apos;d get the following update chain (from oldest to newest):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;RID(2) =&amp;gt; B @ TS(20) -&amp;gt; &amp;lt;tombstone&amp;gt; @ TS(30) -&amp;gt; PreparedInsert @ TS(10)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Committing the prepared insert at a value between 10 and 30 results in wrong results/inconsistent data when reading at those timestamps.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1814107">SERVER-58409</key>
            <summary>Startup RecordId initialization is flawed with durable history and reconstructing prepared transactions</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="daniel.gottlieb@mongodb.com">Daniel Gottlieb</reporter>
                        <labels>
                    </labels>
                <created>Sat, 10 Jul 2021 03:36:09 +0000</created>
                <updated>Sun, 29 Oct 2023 21:51:06 +0000</updated>
                            <resolved>Fri, 29 Oct 2021 21:07:41 +0000</resolved>
                                    <version>5.1.0</version>
                    <version>4.4.6</version>
                    <version>5.0.0-rc8</version>
                                    <fixVersion>5.2.0</fixVersion>
                    <fixVersion>5.1.2</fixVersion>
                    <fixVersion>5.0.6</fixVersion>
                                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>16</watches>
                                                                                                                <comments>
                            <comment id="4225650" author="xgen-internal-githook" created="Wed, 1 Dec 2021 22:24:38 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58409&quot; title=&quot;Startup RecordId initialization is flawed with durable history and reconstructing prepared transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58409&quot;&gt;&lt;del&gt;SERVER-58409&lt;/del&gt;&lt;/a&gt; Use WiredTiger largest_key API to initialize the highest RecordId in a collection&lt;br/&gt;
This fixes a bug that allows a RecordId to be incorrectly reused during startup recovery&apos;s prepared transaction reconstruction.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit c4f813e20892a5bd22d0d7b109542c2b242d4d5e)&lt;br/&gt;
Branch: v5.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c0022ed6effd3419c553597ef575053c80e8aca9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c0022ed6effd3419c553597ef575053c80e8aca9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4224500" author="xgen-internal-githook" created="Wed, 1 Dec 2021 15:59:12 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58409&quot; title=&quot;Startup RecordId initialization is flawed with durable history and reconstructing prepared transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58409&quot;&gt;&lt;del&gt;SERVER-58409&lt;/del&gt;&lt;/a&gt; Use WiredTiger largest_key API to initialize the highest RecordId in a collection&lt;br/&gt;
This fixes a bug that allows a RecordId to be incorrectly reused during startup recovery&apos;s prepared transaction reconstruction.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit c4f813e20892a5bd22d0d7b109542c2b242d4d5e)&lt;br/&gt;
Branch: v5.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/ee678223f9e3ae68a27b3c0f434c2449b903196b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/ee678223f9e3ae68a27b3c0f434c2449b903196b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4158281" author="xgen-internal-githook" created="Fri, 29 Oct 2021 20:30:19 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Louis Williams&apos;, &apos;email&apos;: &apos;louis.williams@mongodb.com&apos;, &apos;username&apos;: &apos;louiswilliams&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-58409&quot; title=&quot;Startup RecordId initialization is flawed with durable history and reconstructing prepared transactions&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-58409&quot;&gt;&lt;del&gt;SERVER-58409&lt;/del&gt;&lt;/a&gt; Use WiredTiger largest_key API to initialize the highest RecordId in a collection&lt;br/&gt;
This fixes a bug that allows a RecordId to be incorrectly reused during startup recovery&apos;s prepared transaction reconstruction.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c4f813e20892a5bd22d0d7b109542c2b242d4d5e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c4f813e20892a5bd22d0d7b109542c2b242d4d5e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4026385" author="louis.williams" created="Fri, 27 Aug 2021 14:04:58 +0000"  >&lt;p&gt;I discussed with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vamsi.krishna&quot; class=&quot;user-hover&quot; rel=&quot;vamsi.krishna&quot;&gt;vamsi.krishna&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=keith.smith&quot; class=&quot;user-hover&quot; rel=&quot;keith.smith&quot;&gt;keith.smith&lt;/a&gt; and we agreed on the solution in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7992&quot; title=&quot;Provide API to return the last key in a table regardless of visibility&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7992&quot;&gt;&lt;del&gt;WT-7992&lt;/del&gt;&lt;/a&gt; to implement the desired &quot;return the last key&quot; behavior.&lt;/p&gt;</comment>
                            <comment id="4017945" author="louis.williams" created="Mon, 23 Aug 2021 19:58:24 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vamsi.krishna&quot; class=&quot;user-hover&quot; rel=&quot;vamsi.krishna&quot;&gt;vamsi.krishna&lt;/a&gt;,&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;my understanding is the read timestamp is of the same transaction which is trying to commit the prepared transaction&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I think you&apos;re mostly describing the correct scenario, but I want to clarify that this prepared transaction is not yet committed and is not in the process of being committed. The prepared transaction is being &quot;reconstructed&quot;, but we have not received an oplog entry to commit yet.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;So, when we get this error, we could roll back the existing transaction and start a new one without any read timestamp and commit the prepared transaction.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I don&apos;t understand why this works. If we retry reconstructing the prepared transaction without a read timestamp, then we will just encounter the same bug that this ticket aims to fix.&lt;/p&gt;

&lt;p&gt;Presently, my proposed code review changes almost 300 lines of code, and it still hasn&apos;t correctly solved this problem. It makes a pretty complex change to our code to dance around the fact that what we really need is an API to return the last key in the table, regardless of visibility. I would like to revisit this idea.&lt;/p&gt;

&lt;p&gt;So to revisit the previous discussion, I&apos;m not certain that the &quot;overwrite=true&quot; case is guaranteed to work in all cases, and I feel that this is really at the wrong entry point to WiredTiger. The ideal solution would be in the read path, not the write path. The other option would be my proposed alternative of adding another assertion suppression in WiredTiger. I&apos;ve filed &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7992&quot; title=&quot;Provide API to return the last key in a table regardless of visibility&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7992&quot;&gt;&lt;del&gt;WT-7992&lt;/del&gt;&lt;/a&gt;. Let me know what you think. CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4013534" author="louis.williams" created="Fri, 20 Aug 2021 16:54:57 +0000"  >&lt;p&gt;Update: I&apos;ve run into some issues with the way we timestamp multikey writes during prepared transaction reconstruction. Specifically, I&apos;m hitting this error code: &quot;commit timestamp _ must be greater than the latest active read timestamp&quot;. I am trying to think through workarounds to this problem, but we are essentially reading at the stable timestamp and committing at the stable timestamp, I don&apos;t think there is a &quot;safe&quot; timestamp to use.&lt;/p&gt;

&lt;p&gt;More specifically:&lt;br/&gt;
MongoDB is performing a write during startup recovery of a prepared transaction where the &lt;tt&gt;read timestamp = prepare timestamp&lt;/tt&gt; case is true. Normally a commit of this prepared transaction would assign a new OpTime, but we have this &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7f4e9553fb617354cabedf6ad6304934e6688efc/src/mongo/db/catalog/index_catalog_entry_impl.cpp#L309-L322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;multikey edge case&lt;/a&gt; where we start a side transaction that tries to perform a write where &lt;tt&gt;commit timestamp = read timestamp&lt;/tt&gt;. The code rounds up the commit timestamp to the stable timestamp, but the stable timestamp is also equal to this timestamp. This is failing the assertion that the commit timestamp is greater than the latest active read timestamp. &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=vamsi.krishna&quot; class=&quot;user-hover&quot; rel=&quot;vamsi.krishna&quot;&gt;vamsi.krishna&lt;/a&gt;/&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chenhao.qu&quot; class=&quot;user-hover&quot; rel=&quot;chenhao.qu&quot;&gt;chenhao.qu&lt;/a&gt;, would it be reasonable to add an additional suppression for this assertion where &quot;round_timestamps=(prepared=true)&quot;?&lt;/p&gt;

&lt;p&gt;One alternative I see is going back to using ghost timestamps (using timestamps further in the past), which are more problematic and incorrect. It would not be possible to just generate a new OpTime because the recovering node is not a primary. &lt;/p&gt;</comment>
                            <comment id="3995082" author="louis.williams" created="Wed, 11 Aug 2021 14:44:30 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, agreed that would be a problem if we default all writes use overwrite=false and also use this new configuration. We could just use this configuration when applying prepared transactions, however, I have some concerns about how we would work this into our current RecordStore API. That would be a messier change for us than just being able to suppress the assertion in WT.&lt;/p&gt;</comment>
                            <comment id="3995004" author="daniel.gottlieb@10gen.com" created="Wed, 11 Aug 2021 14:26:49 +0000"  >&lt;p&gt;Re:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;I think your proposed trick to use prepared timestamp as the read timestamp should work. The error you encountered that &quot;prepared timestamp cannot be smaller or equal to active read timestamp&quot; can be work around. &lt;/p&gt;&lt;/blockquote&gt;

&lt;blockquote&gt;&lt;p&gt;I think you only need to rollback the current transaction if you hit that error&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;We always hit that error. That assertion will always trigger when a single WT transaction reads at X and then tries to prepare at X.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;and then start a new transaction with a larger read timestamp or no read timestamp to reconstruct the prepared update again&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;The algorithm is incorrect if we choose a different (or omit the) read timestamp. We may fail to generate a write conflict and get back into the same situation of creating an out of order update chain.&lt;/p&gt;

&lt;p&gt;Re:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;suggested adding a new config when overwrite=false.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;From what I can tell, our typical record stores use &lt;a href=&quot;https://github.com/mongodb/mongo/blob/a9d88f030f694318e01fec7eeb626503417c8481/src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp#L1480&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;overwrite=true&lt;/a&gt;. I couldn&apos;t say if we actually have to use overwrite=true.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt; After the proposed change, with the config set, we will return conflict if the tombstone is not globally visible (i.e the entry is valid at some point after the oldest timestamp).&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;Assuming we can and want to change these inserts to use overwrite=true, this sounds like it would cause conflicts even when we choose a good timestamp (i.e: false positives for all of oplog recovery). That could be a potentially unbounded cost that everyone would have to pay.&lt;/p&gt;

&lt;p&gt;Does my interpretation match your understanding &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt;?&lt;/p&gt;</comment>
                            <comment id="3994247" author="chenhao.qu" created="Wed, 11 Aug 2021 05:47:51 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; We have reviewed the choices again. We now have two new proposals:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;I think your proposed trick to use prepared timestamp as the read timestamp should work. The error you encountered that &quot;prepared timestamp cannot be smaller or equal to active read timestamp&quot; can be work around. I think you only need to rollback the current transaction if you hit that error and then start a new transaction with a larger read timestamp or no read timestamp to reconstruct the prepared update again. I&apos;m not sure if that is possible or not to rollback that transaction. If that is possible, it should work around that problem.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.gorrod&quot; class=&quot;user-hover&quot; rel=&quot;alexander.gorrod&quot;&gt;alexander.gorrod&lt;/a&gt; suggested adding a new config when overwrite=false. When this new config is set the cursor insert operation will return conflict if the key exists irrespective of its visibility. In the current implementation, if the tombstone is visible, we don&apos;t return conflict for an insert, if overwrite=false. After the proposed change, with the config set, we will return conflict if the tombstone is not globally visible (i.e the entry is valid at some point after the oldest timestamp).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Let us know which way you would like to go.&lt;/p&gt;</comment>
                            <comment id="3992893" author="daniel.gottlieb@10gen.com" created="Tue, 10 Aug 2021 17:12:13 +0000"  >&lt;p&gt;An aside: the reason for using the read timestamp is subtle and clever. This is why I find the other option of getting the (actual) largest key in the table preferable. But more important right now would be expediency. We can always do one solution then switch to the other. &lt;/p&gt;

&lt;p&gt;In my mind, waiting a week for the preferable solution is fine, but if it were to drag out for more than a sprint, I&apos;d opt for the quicker fix.&lt;/p&gt;</comment>
                            <comment id="3992371" author="louis.williams" created="Tue, 10 Aug 2021 14:45:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chenhao.qu&quot; class=&quot;user-hover&quot; rel=&quot;chenhao.qu&quot;&gt;chenhao.qu&lt;/a&gt;, when we reconstruct a prepared insert, we don&apos;t read RID 2 before inserting. We will just blindly insert RID 2 with overwrite=true. In this example, that will write conflict because we&apos;re reading at timestamp 10 and there is an insert at TS 20. We then just retry (after incrementing the RID counter) and blindly insert until the write succeeds with RID 3.&lt;/p&gt;</comment>
                            <comment id="3991636" author="chenhao.qu" created="Tue, 10 Aug 2021 06:57:34 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; We had another look at your proposed solution. We don&apos;t understand how it works. Taking your example, suppose we have done the following operations before restart:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Set OldestTimestamp &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Insert RecordId(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) -&amp;gt; A at TimeStamp(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;10&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Insert RID(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) -&amp;gt; B at TS(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;20&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Delete RID(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;) (B) at TS(&lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;30&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;From my understanding, you will first use timestamp 0 to read the largest key and you should get 1 in this case and decide the next entry should use key 2.&lt;br/&gt;
Then suppose you want to reconstruct a prepared update with timestamp 10, you read key 2 with timestamp 10 again. If that returns a key, then you know there is a deleted key so you need to use key 3 instead of 2 for the new record. However, in your example, if you read at timestamp 10, you will still get not found because the key only exists from timestamp 20 onwards.&lt;/p&gt;

&lt;p&gt;Is my understanding correct? If that is correct, your solution seems not working. If that is not what you mean, can you walk me through your example how everything is designed to work step by step.&lt;/p&gt;</comment>
                            <comment id="3982844" author="louis.williams" created="Thu, 5 Aug 2021 14:17:55 +0000"  >&lt;p&gt;Thanks for the suggestions, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=chenhao.qu&quot; class=&quot;user-hover&quot; rel=&quot;chenhao.qu&quot;&gt;chenhao.qu&lt;/a&gt;. I&apos;ll experiment with this.&lt;/p&gt;</comment>
                            <comment id="3982238" author="chenhao.qu" created="Thu, 5 Aug 2021 05:15:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; I have a new idea to solve this. In WiredTiger cursor, we introduced a flag WT_CURSTD_IGNORE_TOMBSTONE. If you set that flag, it will ignore the deletion of RID 2 when you read the data. Does that solve your issue to find the largest record id? If you think that can work, we can discuss to add a cursor config to set that flag when you open the cursor.&lt;/p&gt;</comment>
                            <comment id="3979115" author="louis.williams" created="Tue, 3 Aug 2021 20:39:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, this makes sense to me. I just tested this algorithm with the reproducer and this seems to work. I opened &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7918&quot; title=&quot;Allow setting the prepare timestamp smaller than or equal to the latest active read timestamp with roundup prepare config&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7918&quot;&gt;&lt;del&gt;WT-7918&lt;/del&gt;&lt;/a&gt; to track the WiredTiger request.&lt;/p&gt;</comment>
                            <comment id="3978315" author="daniel.gottlieb@10gen.com" created="Tue, 3 Aug 2021 15:42:33 +0000"  >&lt;blockquote&gt;
&lt;p&gt;Say we prepared an insert at TS 10 that is then reconstructed after a restart. We use our new proposal to read at TS 10, this initializes the highest seen RID to 1,&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;That&apos;s not the algorithm I was envisioning. I was thinking:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;On startup, initialize the RecordId for each collection as we do today, by using a non-timestamped read + prev&apos;ing the cursor (I believe the lazy loading is inconsequential). This gets the highest record id in use at &quot;now&quot;.&lt;/li&gt;
	&lt;li&gt;When reconstructing prepared transactions, assign a read timestamp in an effort to force a conflict when the prepare range overlaps with a delete that&apos;s timestamped between the oldest and stable timestamps (when applicable).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Using your example, the next record id for an insert would be initialized to 4. IIUC, that would be a suitable key to prepare the insert with.&lt;/p&gt;

&lt;p&gt;Does the more detailed algorithm still seem problematic?&lt;/p&gt;</comment>
                            <comment id="3978288" author="louis.williams" created="Tue, 3 Aug 2021 15:32:41 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, I&apos;ve been thinking over this problem, and I think the proposed fix just defers the same problem past the prepared transaction reconstruction phase. For example. Say there is a collection with update chains like this:&lt;/p&gt;

&lt;p&gt;RID 1: A @ TS(5) -&amp;gt; Tombstone @ TS(15)&lt;br/&gt;
RID 2: deleted &lt;br/&gt;
RID 3: B @ TS(20)&lt;/p&gt;

&lt;p&gt;Say we prepared an insert at TS 10 that is then reconstructed after a restart. We use our new proposal to read at TS 10, this initializes the highest seen RID to 1, and then inserts the new record at RID 2. If a new operation inserts a record (read TS &amp;gt;= 20), it will insert a new record at RID 3. This conflicts with an existing record. Because MongoDB configures cursors to &quot;overwrite=true&quot;, this overwrites an existing record. I can introduce this failure with a slightly modified test: &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/328228/328228_server-58409.js&quot; title=&quot;server-58409.js attached to SERVER-58409&quot;&gt;server-58409.js&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;.&lt;/p&gt;</comment>
                            <comment id="3975821" author="daniel.gottlieb@10gen.com" created="Mon, 2 Aug 2021 16:18:13 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams&quot;&gt;louis.williams&lt;/a&gt; and I were unable to formulate a non-WT change to get around the assertion. However, we have the following patch that suppresses that assertion when using the existing WT flag for rounding up a prepared transactions timestamp to the oldest timestamp (only used for reconstructing prepared transactions):&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;diff --git a/src/third_party/wiredtiger/src/txn/txn_timestamp.c b/src/third_party/wiredtiger/src/txn/txn_timestamp.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;index 9fc79a78d7..b290fee7dd 100644&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;--- a/src/third_party/wiredtiger/src/txn/txn_timestamp.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+++ b/src/third_party/wiredtiger/src/txn/txn_timestamp.c&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;@@ -714,7 +714,8 @@ __wt_txn_set_prepare_timestamp(WT_SESSION_IMPL *session, wt_timestamp_t prepare_&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;         WT_RET_MSG(session, EINVAL,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;           &quot;commit timestamp should not have been set before the prepare timestamp&quot;);&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-    WT_RET(__txn_assert_after_reads(session, &quot;prepare&quot;, prepare_ts));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+    if (!F_ISSET(txn, WT_TXN_TS_ROUND_PREPARED))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;+        WT_RET(__txn_assert_after_reads(session, &quot;prepare&quot;, prepare_ts));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;     /*&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;      * Check whether the prepare timestamp is less than the oldest timestamp.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="3973381" author="louis.williams" created="Fri, 30 Jul 2021 18:01:34 +0000"  >&lt;p&gt;Option 3 doesn&apos;t work because of the WiredTiger assertion: &quot;prepare timestamp X must be greater than the latest active read timestamp Y&quot;&lt;/p&gt;</comment>
                            <comment id="3926965" author="daniel.gottlieb@10gen.com" created="Sat, 10 Jul 2021 18:12:08 +0000"  >&lt;p&gt;A couple less attractive solutions followed by a clever trick?&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Have WT expose an API, e.g: &lt;tt&gt;WT_SESSION::getLargestKeyForTable(char* tableName, vararg *keyOutParam)&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;At startup, instead of &quot;get largest record id in current (i.e: stable timestamp) snapshot&quot; do:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;for each timestamp between oldest timestamp and stable timestamp:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  ret = max(ret, getLargestRecordIdAt(timestamp));&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;When reconstructing prepared transactions at startup/after rollback, set the read timestamp to the prepared timestamp with roundToOldest=true. On WCE, retry the operation (instead of presumably crashing today). I expect the retry to naturally try inserting with the next available record id until the reconstruction succeeds.&lt;/li&gt;
&lt;/ol&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1900787">WT-8241</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1895478">WT-8226</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1840145">WT-7918</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1858374">WT-7992</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1876752">WT-8114</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="1964192">SERVER-62650</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1808526">WT-7783</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1813195">WT-7815</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1814066">WT-7820</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="324460" name="25686_repro.js" size="3601" author="daniel.gottlieb@mongodb.com" created="Sat, 10 Jul 2021 03:38:02 +0000"/>
                            <attachment id="328228" name="server-58409.js" size="3894" author="louis.williams@mongodb.com" created="Tue, 3 Aug 2021 15:32:10 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>20.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="22495"><![CDATA[v5.1]]></customfieldvalue>
    <customfieldvalue key="21777"><![CDATA[v5.0]]></customfieldvalue>
    <customfieldvalue key="18953"><![CDATA[v4.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 30 Jul 2021 18:01:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 10 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/WT-7918'>WT-7918</a></s>, <s><a href='https://jira.mongodb.org/browse/WT-7992'>WT-7992</a></s>, <s><a href='https://jira.mongodb.org/browse/WT-8226'>WT-8226</a></s>, <s><a href='https://jira.mongodb.org/browse/WT-8241'>WT-8241</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 10 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>135.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>chenhao.qu@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzrgbj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2wrb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5116">Execution Team 2021-08-09</customfieldvalue>
    <customfieldvalue id="5117">Execution Team 2021-08-23</customfieldvalue>
    <customfieldvalue id="5118">Execution Team 2021-09-06</customfieldvalue>
    <customfieldvalue id="5119">Execution Team 2021-09-20</customfieldvalue>
    <customfieldvalue id="5382">Execution Team 2021-10-18</customfieldvalue>
    <customfieldvalue id="5420">Execution Team 2021-11-01</customfieldvalue>
    <customfieldvalue id="5421">Execution Team 2021-11-15</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzr2kn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>