<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:31:27 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32843] OpObserver&apos;s callback methods must not have return values</title>
                <link>https://jira.mongodb.org/browse/SERVER-32843</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The OpObserverRegistry was introduced as an abstraction to allow decoupling making data modifications from the side effects, which need to happen as a result of these modifications, such as op log writes, retryable writes, etc.&lt;/p&gt;

&lt;p&gt;Some of the OpObserver&apos;s methods currently return the OpTime which resulted from logging the operation to the replication oplog. In addition, in certain cases, the OpTime resulting from an earlier OpObserver might be needed by a later one, which is the case with retryable writes.&lt;/p&gt;

&lt;p&gt;In order to support these requirements, the OpObserver(s) chain should have access to some common per-operation structure, where runtime information could be persisted.&lt;/p&gt;

&lt;p&gt;The specification is:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Add a decoration on the `OperationContext` and place it in the protected section of `OpObserver`. It should contain an `std::vector&amp;lt;OpTime&amp;gt; reservedOpTimes`.&lt;/li&gt;
	&lt;li&gt;Change all `OpObserverRegistry` methods to have a scoped guard, which clears that vector.&lt;/li&gt;
	&lt;li&gt;Change the `repl::logOp` call to invariant that this vector is empty and populate it with the set of timestamps, which it produced.&lt;/li&gt;
	&lt;li&gt;Change the `OpObserverRegistry&apos;s onDropCollection` and `onRenameCollection` methods to not call `_forEachObserver`, but instead loop and invariant that the return values of the chained calls are null `OpTimes` (`isNull()`) and instead end in:&lt;br/&gt;
        `if (reservedOpTimes.empty()) return OpTime();&lt;br/&gt;
        else 
{ invariant(reservedOpTimes.size() == 1); return reservedOpTimes[0];}
&lt;p&gt;`&lt;br/&gt;
    Change `OpObserverImpl`&apos;s `onDropCollection` and `onRenameCollection` methods to return null `OpTimes`.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
                <environment></environment>
        <key id="485900">SERVER-32843</key>
            <summary>OpObserver&apos;s callback methods must not have return values</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="adam.martin@mongodb.com">ADAM Martin</assignee>
                                    <reporter username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</reporter>
                        <labels>
                    </labels>
                <created>Mon, 22 Jan 2018 21:15:45 +0000</created>
                <updated>Mon, 30 Oct 2023 23:09:06 +0000</updated>
                            <resolved>Thu, 1 Mar 2018 21:53:20 +0000</resolved>
                                                    <fixVersion>3.7.3</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1820477" author="xgen-internal-githook" created="Thu, 1 Mar 2018 21:50:22 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;adam.martin@10gen.com&apos;, &apos;name&apos;: &apos;ADAM David Alan Martin&apos;, &apos;username&apos;: &apos;adamlsd&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32843&quot; title=&quot;OpObserver&amp;#39;s callback methods must not have return values&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32843&quot;&gt;&lt;del&gt;SERVER-32843&lt;/del&gt;&lt;/a&gt; Allow multiple times in OpObservers&lt;/p&gt;

&lt;p&gt;The OpObserverRegistry was introduced as an abstraction to allow&lt;br/&gt;
decoupling making data modifications from the side effects, which&lt;br/&gt;
need to happen as a result of these modifications, such as op log&lt;br/&gt;
writes, retryable writes, etc.&lt;/p&gt;

&lt;p&gt;Some of the OpObserver&apos;s methods currently return the OpTime which&lt;br/&gt;
resulted from logging the operation to the replication oplog. In&lt;br/&gt;
addition, in certain cases, the OpTime resulting from an earlier&lt;br/&gt;
OpObserver might be needed by a later one, which is the case with&lt;br/&gt;
retryable writes.&lt;/p&gt;

&lt;p&gt;In order to support these requirements, the OpObserver(s) chain&lt;br/&gt;
should have access to some common per-operation structure, where&lt;br/&gt;
runtime information could be persisted.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/296fde1259d29e081069fde1c69bb9ae083932b1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/296fde1259d29e081069fde1c69bb9ae083932b1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1785329" author="spencer" created="Thu, 25 Jan 2018 20:57:26 +0000"  >&lt;p&gt;lgtm&lt;/p&gt;</comment>
                            <comment id="1785319" author="kaloian.manassiev" created="Thu, 25 Jan 2018 20:50:21 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt; to your questions - I just updated the &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32843?focusedCommentId=1785038&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1785038&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;comment&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1785282" author="spencer" created="Thu, 25 Jan 2018 20:27:32 +0000"  >&lt;p&gt;I don&apos;t think repl::logOp should be responsible for clearing the vector, I think that should be done by the OpObserverRegistry after running the last observer and saving aside whatever it needs to return.  repl::logOp should invariant that the vector is empty when it starts.&lt;/p&gt;

&lt;p&gt;I also don&apos;t like the term &apos;uncommittedOpTimes&apos; as the term &apos;uncommitted&apos; can mean different things at different layers.  I&apos;d prefer either &apos;reservedOpTimes&apos;, &apos;generatedOpTimes&apos;, or something similar.&lt;/p&gt;</comment>
                            <comment id="1785100" author="judah.schvimer" created="Thu, 25 Jan 2018 18:33:07 +0000"  >&lt;p&gt;So all OpObservers that the registry calls will return OpTime() for onDropCollection and onRenameCollection?&lt;/p&gt;</comment>
                            <comment id="1785038" author="kaloian.manassiev" created="Thu, 25 Jan 2018 18:02:41 +0000"  >&lt;p&gt;The final specification we came up with is:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Add a decoration on the OperationContext and place it in the protected section of OpObserver. It should contains an &lt;tt&gt;std::vector&amp;lt;OpTime&amp;gt; reservedOpTimes&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;Change all &lt;tt&gt;OpObserverRegistry&lt;/tt&gt; methods to have a scoped guard, which clears that vector.&lt;/li&gt;
	&lt;li&gt;Change the &lt;a href=&quot;https://github.com/mongodb/mongo/blob/048da5de50a2f6aa8776f1ecd626313f5ca1e575/src/mongo/db/repl/oplog.cpp#L435&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;repl::logOp&lt;/tt&gt; call&lt;/a&gt; to invariant that this vector is empty and populate it with the set of timestamps, which it produced.&lt;/li&gt;
	&lt;li&gt;Change the OpObserverRegistry&apos;s &lt;tt&gt;onDropCollection&lt;/tt&gt; and &lt;tt&gt;onRenameCollection&lt;/tt&gt; methods to not call &lt;tt&gt;_forEachObserver&lt;/tt&gt;, but use loop and invariant that the return values of the chained calls are null OpTimes (isNull()) and instead end in:
	&lt;ul&gt;
		&lt;li&gt;&lt;tt&gt;if (reservedOpTimes.empty()) return OpTime();&lt;/tt&gt;&lt;/li&gt;
		&lt;li&gt;&lt;tt&gt;else { invariant(reservedOpTimes.size() == 1); return reservedOpTimes[0];&lt;/tt&gt;}&lt;/li&gt;
	&lt;/ul&gt;
	&lt;/li&gt;
	&lt;li&gt;Change OpObserverImpl&apos;s  &lt;tt&gt;onDropCollection&lt;/tt&gt; and &lt;tt&gt;onRenameCollection&lt;/tt&gt; methods to return null OpTimes.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1784918" author="spencer" created="Thu, 25 Jan 2018 16:42:48 +0000"  >&lt;p&gt;The overall approach sounds good, though I&apos;m not sold on the name TransactionContext.  Counterproposal: how about we just decorate the OperationContext directly with the vector of OpTimes, and call that vector &apos;reservedOpTimes&apos;?&lt;/p&gt;

&lt;p&gt;For this to work we&apos;ll have to ensure that whatever OpObserver generates the OpTimes (currently OpObserverImpl, but likely to soon be a separate ReplicationOpObserver) runs first.  I think we can do this by making sure the OpObserverRegistry guarantees that it will run observers in the order they are registered, and make sure to always register the observer that calls logOp first.&lt;/p&gt;</comment>
                            <comment id="1784746" author="kaloian.manassiev" created="Thu, 25 Jan 2018 15:11:38 +0000"  >&lt;p&gt;It is the latter, yes.&lt;/p&gt;

&lt;p&gt;If we can define that one invocation of the OpObserver chain belongs to one &quot;logical write&quot; and than one WUOW can have multiple &quot;logical writes&quot;, we need OpObservers later in the chain to access the OpTimes that OpObservers earlier in the chain generated for the same &quot;logical write&quot;.&lt;/p&gt;

&lt;p&gt;I am saying OpTimes (in plural) because onInserts for example generates more than one OpTime and if are to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f6d5c6acf7481833b0e0cd579f9145e4f21b0ace/src/mongo/db/op_observer_impl.cpp#L330&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;move the CSS&lt;/a&gt; to a separate OpObserver, it will need to access all of them.&lt;/p&gt;

&lt;p&gt;Based on these requirements, does the proposal in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32843?focusedCommentId=1781356&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1781356&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this comment&lt;/a&gt; sound reasonable?&lt;/p&gt;</comment>
                            <comment id="1784270" author="spencer" created="Wed, 24 Jan 2018 23:56:03 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev&quot;&gt;kaloian.manassiev&lt;/a&gt;, are you saying that an OpObserver for a later write needs the optime(s) generated by an OpObserver for an earlier write?  Or just that later OpObservers in the chain of registered OpObservers for a single write might need access to the OpTime of the write it is handling?&lt;/p&gt;

&lt;p&gt;The latter makes sense to me, as it&apos;s something that the current handlers in the single OpObserver we have today have access to.  If it&apos;s the former I&apos;d like to understand better why it&apos;s needed.&lt;/p&gt;</comment>
                            <comment id="1782435" author="matthew.russotto" created="Tue, 23 Jan 2018 17:44:52 +0000"  >&lt;p&gt;The complication is we can have multiple OpObserver calls within the same write unit of work.  If the later OpObservers need context that is specifically about a single OpObserver call (and I think they do), this wouldn&apos;t do what we want. &lt;/p&gt;

&lt;p&gt;We do need something which records uncommitted changes for a transaction, but that needs to go elsewhere (something hung off the Session object is what the design says now)&lt;/p&gt;</comment>
                            <comment id="1782348" author="kaloian.manassiev" created="Tue, 23 Jan 2018 17:03:10 +0000"  >&lt;p&gt;Yes, this new decoration is for passing OpTimes (or other state) from one OpObserver call to the next and it will be essentially a stack of all OpLog entries, which happened as part of one WUOW.&lt;/p&gt;

&lt;p&gt;Perhaps it should be called WriteUnitOfWorkContext then? It should be empty when WUOW begins, then OpObserver calls append to it and at commit time it is reset back to empty. How does that sound? Is this something also which would be useful for tracking the in-progress transaction state, because this is essentially what this is - a log of the uncommitted changes for a transaction?&lt;/p&gt;</comment>
                            <comment id="1782303" author="matthew.russotto" created="Tue, 23 Jan 2018 16:39:24 +0000"  >&lt;p&gt;I like the idea of putting it on the OperationContext rather the ReplClientInfo.  I&apos;m not sure it&apos;s sufficient for passing opTimes from opObserver to opObserver (is that what this is for?).  Reason being there&apos;s no rule saying an opObserver be called only once in an operation context, so you wouldn&apos;t know where to start.  It might be necessary to explicitly have some object hoding state for the opObserver calls.&lt;/p&gt;</comment>
                            <comment id="1781356" author="kaloian.manassiev" created="Mon, 22 Jan 2018 21:25:58 +0000"  >&lt;p&gt;I propose to introduce a decoration on the OperationContext called &lt;tt&gt;TransactionContext&lt;/tt&gt;, which contains the following field:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;b&gt;std::vector&amp;lt;OpTime&amp;gt; uncommittedOpTimes&lt;/b&gt; - the set of op times which have been generated as part of this OpObserver chain invocation. In most of the cases, this field will contain only one element, but it needs to be a vector for the purposes of &lt;tt&gt;onInserts&lt;/tt&gt;, which can generate more than one OpLog entry.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The &lt;tt&gt;logOp&lt;/tt&gt; call will be changed to look for the OpObserver context and if there is one will append the timestamps it generates to it. At transaction commit time, the &lt;tt&gt;ReplClientInfo&lt;/tt&gt;&apos;s lastCommitted time will be advanced to the highest of the uncommittedOpTimes (which should be the latest).&lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=matthew.russotto&quot; class=&quot;user-hover&quot; rel=&quot;matthew.russotto&quot;&gt;matthew.russotto&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="487345">SERVER-32897</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 23 Jan 2018 16:39:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 49 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-331</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 49 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>adam.martin@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>matthew.russotto@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htoidj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htg35r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2093">Platforms 2018-01-29</customfieldvalue>
    <customfieldvalue id="2129">Platforms 2018-02-12</customfieldvalue>
    <customfieldvalue id="2130">Platforms 2018-02-26</customfieldvalue>
    <customfieldvalue id="2131">Platforms 2018-03-12</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hto4hz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>