<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:45:58 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-58965] Provide an interface to access replication information</title>
                <link>https://jira.mongodb.org/browse/SERVER-58965</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;In the User-facing Point-In-Time pre- and post-images project (PM-1944) there is a need for accessing replication related information in&#160;OpObserverImpl class in order to be able to correctly maintain pre-image documents:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;whether the documents are in &lt;a href=&quot;https://github.com/mongodb/mongo/blob/49f9319d0ee50502259b61dfef2e2b71bf86948a/src/mongo/db/repl/oplog.cpp#L1102-L1103&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;consistent state &lt;/a&gt;;&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Please provide an interface that gives an access to listed above information.&lt;/p&gt;
</description>
                <environment></environment>
        <key id="1836404">SERVER-58965</key>
            <summary>Provide an interface to access replication information</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="matthew.russotto@mongodb.com">Matthew Russotto</assignee>
                                    <reporter username="mindaugas.malinauskas@mongodb.com">Mindaugas Malinauskas</reporter>
                        <labels>
                    </labels>
                <created>Fri, 30 Jul 2021 09:40:56 +0000</created>
                <updated>Fri, 27 Oct 2023 20:45:57 +0000</updated>
                            <resolved>Tue, 14 Sep 2021 14:53:18 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="4058199" author="matthew.russotto" created="Tue, 14 Sep 2021 14:53:18 +0000"  >&lt;p&gt;Spoke to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mindaugas.malinauskas&quot; class=&quot;user-hover&quot; rel=&quot;mindaugas.malinauskas&quot;&gt;mindaugas.malinauskas&lt;/a&gt;, we currently believe this ticket is not necessary.&lt;/p&gt;</comment>
                            <comment id="4024478" author="daniel.gottlieb@10gen.com" created="Thu, 26 Aug 2021 15:55:56 +0000"  >&lt;p&gt;(edit) While what I said was true, I was misinterpreting Matthew&apos;s comment. I&apos;m also of the understanding that having change streams (with preImages or otherwise) and tenant migrations play nicely together isn&apos;t in scope.&lt;/p&gt;

&lt;p&gt;&lt;del&gt;Tenant migrations sets &quot;enforcing constraints&quot; to false as well as using false for &quot;isDataConsistent&quot; when calling into oplog application. I don&apos;t think OpObserverImpl has any more  context than oplog application.&lt;/del&gt;&lt;br/&gt;
 &lt;a href=&quot;https://github.com/mongodb/mongo/blob/ea63938df2478ad111421b847818fc64b00dedca/src/mongo/db/repl/tenant_oplog_applier.cpp#L1013-L1027&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/ea63938df2478ad111421b847818fc64b00dedca/src/mongo/db/repl/tenant_oplog_applier.cpp#L1013-L1027&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4024216" author="matthew.russotto" created="Thu, 26 Aug 2021 14:49:40 +0000"  >&lt;p&gt;Tenant migration works like initial sync, in that we do oplog application on an inconsistent view of the data to bring it to a consistent view.&#160; Difference is it does it on the primary.&#160; The OpObserverImpl can know when it&apos;s doing oplog application in tenant migration (there&apos;s a decorator on the OpContext.&#160; However, on the secondary that decorator won&apos;t be available.&lt;/p&gt;</comment>
                            <comment id="4023143" author="daniel.gottlieb@10gen.com" created="Thu, 26 Aug 2021 02:09:08 +0000"  >&lt;blockquote&gt;
&lt;p&gt;I think checking isEnforcingConstraints() could work well for us here. Would that work on both primaries and secondaries? For what scenarios that does return false?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&lt;tt&gt;isEnforcingConstraints&lt;/tt&gt; returns false during oplog application. It doesn&apos;t distinguish between the states of interest:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Applying oplog entries on a consistent set of data (steady state oplog application) &amp;#8211; record preimages&lt;/li&gt;
	&lt;li&gt;Applying oplog entries during initial sync (or the now officially obsolete rollback via refetch?) &amp;#8211; do not record preimages&lt;/li&gt;
&lt;/ul&gt;


&lt;blockquote&gt;
&lt;p&gt;Eric Milkie suggested an alternative in which only OpObserverImpl is responsible for writing pre-images depending on the context it is run&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;What Eric suggested is a perfectly valid choice and I can see the appeal. That said, with the retryable find and modify project to implicitly replicate/record preimages recently done, I think saving the preImages during oplog application explicitly inside oplog.cpp has the following advantages:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;there&apos;s less typing to do overall&lt;/li&gt;
	&lt;li&gt;the code logic is better self-contained (based off of my guess of how things shake out, the thing in oplog.cpp that says &quot;have this update save the preImage&quot; is also the code that consumes the returned preImage and writes it to the change streams preImage collection).&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;But, if we stick with using OpObserverImpl, two ideas:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Thread the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47c2829b6af290584ae083720e73a6ad8376bbc4/src/mongo/db/repl/oplog.cpp#L1100&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;isDataConsistent from oplog.cpp&lt;/a&gt; through the UpdateRequest through &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47c2829b6af290584ae083720e73a6ad8376bbc4/src/mongo/db/op_observer_impl.cpp#L573&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;OplogUpdateEntryArgs&lt;/a&gt;. This is maybe as simple as making sure &lt;tt&gt;UpdateRequest::setReturnDocs&lt;/tt&gt; &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47c2829b6af290584ae083720e73a6ad8376bbc4/src/mongo/db/repl/oplog.cpp#L1452-L1454&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;is left unset&lt;/a&gt;?&lt;/li&gt;
	&lt;li&gt;Write incorrect preimages to the collection anyways. When servicing change stream results, use something like the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/47c2829b6af290584ae083720e73a6ad8376bbc4/src/mongo/db/repl/initial_syncer.cpp#L556&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&quot;initial data timestamp&quot; that initial sync sets&lt;/a&gt; to determine whether a preImage should be returned (preImage timestamp &amp;lt;= initial data timestamp are invalid). This won&apos;t be sufficient if features like tenant migrations need to be dealt with (I&apos;m unsure how it clones data, I just know it works around some forms of inconsistency inside oplog application).&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="4022246" author="milkie" created="Wed, 25 Aug 2021 17:06:50 +0000"  >&lt;p&gt;I think checking isEnforcingConstraints() could work well for us here.  Would that work on both primaries and secondaries? For what scenarios that does return false?&lt;/p&gt;</comment>
                            <comment id="4021670" author="JIRAUSER1252520" created="Wed, 25 Aug 2021 13:46:29 +0000"  >&lt;p&gt;Maybe my reference to &quot;isDataConsistent&quot; introduced some confusion. I&apos;m not 100% sure only this variable encodes the state we are interested in. &lt;/p&gt;

&lt;p&gt;In &lt;a href=&quot;https://jira.mongodb.org/browse/PM-1944&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;PM-1944&lt;/a&gt; we would like to be able to determine if a write operation (update or delete) operates on a document version that is the same as it was when the same operation was executed on the primary node. We need this to avoid preserving an incorrect pre-image of a document when we cannot ensure correctness - one scenario when this can happen is initial sync process.&lt;/p&gt;

&lt;p&gt;I initially worked on a design that was similar to what has been done for PM-2213 and similar to your proposal in the comment above. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; suggested an alternative in which only OpObserverImpl is responsible for writing pre-images depending on the context it is run. To determine the context OpObserverImpl would need some means to learn about &quot;data consistency&quot; status.&lt;/p&gt;

&lt;p&gt;CC: &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matthew.russotto&quot; class=&quot;user-hover&quot; rel=&quot;matthew.russotto&quot;&gt;matthew.russotto&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3976089" author="daniel.gottlieb@10gen.com" created="Mon, 2 Aug 2021 17:55:00 +0000"  >&lt;p&gt;&quot;isDataConsistent&quot; means a specific thing during oplog application that (if the same words were used) wouldn&apos;t be true at the OpObserverImpl layer. Specifically, oplog application starting with consistent data at T and applying changes up to T + 10 will apply them out of order. For example, T + 9 may be applied before any of T -&amp;gt; T + 8. If code in OpObserverImpl asked &quot;is data consistent&quot; in this replication state, there are two valid answers:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Yes, the existing data constrained exactly to the documents being touched are correct.&lt;/li&gt;
	&lt;li&gt;No, causal dependencies of this write may not currently be visible.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Inside oplog.cpp, the second interpretation isn&apos;t any more true, but it is assumed anyone touching that code knows about the relaxed definition of consistency. I imagine for what PM-1944 wants, only the first criteria matters (the document being updated has all previous updates applied), which would be true for this case.&lt;/p&gt;

&lt;p&gt;With PM-2213, we had to solve a similar problem (undoubtedly how you came across that variable inside oplog application). We instead leveraged the fact that a primary accepting writes via a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/93fd56bef8e72ab260a5338deeaf463f5b871099/src/mongo/db/commands/find_and_modify.cpp#L167-L168&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;findAndModify&lt;/a&gt; must be operating on &quot;consistent&quot; data and &lt;a href=&quot;https://github.com/mongodb/mongo/blob/93fd56bef8e72ab260a5338deeaf463f5b871099/src/mongo/db/op_observer_impl.cpp#L622-L634&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;may persist a pre/post image&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Oplog application &lt;a href=&quot;https://github.com/mongodb/mongo/blob/93fd56bef8e72ab260a5338deeaf463f5b871099/src/mongo/db/repl/oplog.cpp#L256&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;has its own codepath&lt;/a&gt; for writing these pre/post images.&lt;/p&gt;

&lt;p&gt;All said, I&apos;m a bit hesitant to exposing this state. I&apos;d prefer for PM-1944 to explore a solution that is similar to PM-2213. Specifically:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;A primary maintaining pre-image documents would annotate incoming commands with some &quot;save the preimage&quot; state on CollectionUpdateArgs/OplogDeleteEntryArgs
	&lt;ul&gt;
		&lt;li&gt;If that&apos;s too much surface area &amp;#8211; perhaps having OpObserverImpl checking for &lt;a href=&quot;https://github.com/mongodb/mongo/blob/b9a5d279a93e8379e189e9d74a0bdfa93340b6fe/src/mongo/db/operation_context.h#L467-L469&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;OperationContext::isEnforcingConstraints&lt;/a&gt; can be used (as constraints can only be enforced if both definitions of consistency are true).&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;A secondary maintaining pre-image documents would explicitly make their writes as part of oplog application. That codepath would get into OpObserverImpl as well, but wouldn&apos;t see the state set on incoming commands (or &lt;tt&gt;isEnforcingConstraints() == false&lt;/tt&gt; if that was the how things were done).&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 2 Aug 2021 17:55:00 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 21 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1944</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 21 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
            <customfieldvalue>mindaugas.malinauskas@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzv8s7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzfh9j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5142">Repl 2021-09-06</customfieldvalue>
    <customfieldvalue id="5143">Repl 2021-09-20</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzuv1b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>