<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:51:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-39356] Change stream updateLookup queries with speculative majority may return uncommitted data</title>
                <link>https://jira.mongodb.org/browse/SERVER-39356</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Speculative majority change streams that do an update lookup query will wait for the most recent lastApplied optime of a replica set node to majority commit before returning results to the client. This is intended to provide a guarantee to the client that the data it received is majority committed. This contract may be violated, however, in the case where a node&apos;s &lt;tt&gt;lastApplied&lt;/tt&gt; optime lags behind the optime of the newest &quot;storage committed&quot; oplog entry. That is, there may be an oplog entry (and corresponding data operation) written to storage that is visible to readers, but the lastApplied optime of the node does not yet reflect it. This is possible because a primary node advances its lastApplied optime inside the onCommit handler of an operation&apos;s transaction. There is a nonzero length of time between the commit of the WriteUnitOfWork at the storage layer and when the optime is advanced for that operation. If a concurrent reader reads the effects of such a transaction and reads lastApplied before the onCommit handler has fired, it may wait for the incorrect optime to commit and return data that is not, in fact, majority committed. This is an issue for primaries. On secondaries lastApplied is only updated at the end of batch application, so the same problem does not manifest. &lt;/p&gt;</description>
                <environment></environment>
        <key id="682368">SERVER-39356</key>
            <summary>Change stream updateLookup queries with speculative majority may return uncommitted data</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Fri, 1 Feb 2019 23:57:38 +0000</created>
                <updated>Sun, 29 Oct 2023 22:24:27 +0000</updated>
                            <resolved>Fri, 15 Mar 2019 18:14:05 +0000</resolved>
                                    <version>4.1.7</version>
                                    <fixVersion>4.1.10</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2184404" author="xgen-internal-githook" created="Mon, 18 Mar 2019 21:34:08 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39383&quot; title=&quot;Speculative majority change stream update lookup on secondary may return pre-image &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39383&quot;&gt;&lt;del&gt;SERVER-39383&lt;/del&gt;&lt;/a&gt; Add a test for speculative majority change stream secondary reads during batch application&lt;/p&gt;

&lt;p&gt;This commit adds an integration test to verify that speculative majority change stream reads do not return incorrect results when reading concurrently with secondary batch application. The goal is to ensure that, due to the changes from &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39356&quot; title=&quot;Change stream updateLookup queries with speculative majority may return uncommitted data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39356&quot;&gt;&lt;del&gt;SERVER-39356&lt;/del&gt;&lt;/a&gt;, these reads will read from the most recent lastApplied timestamp on secondaries.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c8120ddaf8a8bd9da9c8095165a4df485d5a58c9&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c8120ddaf8a8bd9da9c8095165a4df485d5a58c9&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2182020" author="xgen-internal-githook" created="Fri, 15 Mar 2019 18:13:39 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;name&apos;: &apos;William Schultz&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39356&quot; title=&quot;Change stream updateLookup queries with speculative majority may return uncommitted data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39356&quot;&gt;&lt;del&gt;SERVER-39356&lt;/del&gt;&lt;/a&gt; Make speculative majority change stream reads utilize the `kNoOverlap` timestamp read source&lt;/p&gt;

&lt;p&gt;Speculative majority change streams provide &quot;majority&quot; read guarantees by reading from a local snapshot of data and then waiting for that data to become majority committed, instead of reading directly from a majority committed snapshot. In order to satisfy this guarantee a speculative majority read must wait for the proper timestamp to become majority committed after reading data. If the newest data it read reflects a timestamp T, then it must wait for a timestamp &amp;gt;= T to become majority committed. In general, waiting on replication&apos;s lastApplied timestamp is not safe, since it is possible for writes to be visible to readers even if those writes have not yet advanced the in-memory value of lastApplied. To work around this issue for speculative majority reads, we instead choose to read from an explicitly chosen timestamp in the storage engine, and then wait on that timestamp to majority commit. This gives us a more direct way to know what timestamp the data we read reflects. We utilize the `kNoOverlap` read source, which allows us to read from the min(lastApplied, all_committed), which is a convenient way to make these reads work correctly on both primaries and secondaries.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/c83b8d8aab53f7545851a76425b2f2cd7c598cbd&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/c83b8d8aab53f7545851a76425b2f2cd7c598cbd&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2173384" author="xgen-internal-githook" created="Wed, 6 Mar 2019 21:42:39 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;William Schultz&apos;, &apos;email&apos;: &apos;william.schultz@mongodb.com&apos;, &apos;username&apos;: &apos;will62794&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39356&quot; title=&quot;Change stream updateLookup queries with speculative majority may return uncommitted data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39356&quot;&gt;&lt;del&gt;SERVER-39356&lt;/del&gt;&lt;/a&gt; Refactor speculative majority read data structures and methods to use timestamps instead of optimes&lt;/p&gt;

&lt;p&gt;This patch refactors the SpeculativeMajorityReadInfo class and the awaitOpTimeCommitted method to accept timestamps as input instead of optimes. When waiting for an operation to majority commit, term information, which is included in optimes, isn&apos;t necessary, since timestamps are totally ordered within a local oplog, and so are safely comparable. It is, for example, safe to determine if a local oplog entry is majority committed by checking if its timestamp is less than that node&apos;s local view of the majority commit point. This patch should not introduce any observable functional changes.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/fcbc0c9c936c83612545ee5873b649854e4b5e57&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/fcbc0c9c936c83612545ee5873b649854e4b5e57&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2138947" author="william.schultz" created="Tue, 5 Feb 2019 22:11:41 +0000"  >&lt;p&gt;This problem and intended fix should be specific to primaries. The proposed solution is to have update lookup queries read from all_committed on primary and wait on that timestamp to commit so we can be guaranteed the data read became committed. On secondaries, the issue is not quite the same, since lastApplied is only updated at the end of each batch. On secondaries, update lookup queries can read during the middle of batch application, which causes a different problem, referenced in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-39383&quot; title=&quot;Speculative majority change stream update lookup on secondary may return pre-image &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-39383&quot;&gt;&lt;del&gt;SERVER-39383&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="702578">SERVER-39753</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="684317">SERVER-39383</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1756787">SERVER-57197</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="683182">SERVER-39364</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 4 Feb 2019 14:34:30 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 47 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1191</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 47 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>57.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hukzmn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|huauzb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2784">Repl 2019-02-25</customfieldvalue>
    <customfieldvalue id="2822">Repl 2019-03-11</customfieldvalue>
    <customfieldvalue id="2823">Repl 2019-03-25</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huklvz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>