<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:45:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-37178] [POC] Allow change streams to work with &quot;blocking&quot; implementation of majority reads</title>
                <link>https://jira.mongodb.org/browse/SERVER-37178</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We would like to investigate and prototype a solution for implementing change streams correctly without storage engine support for majority reads. The idea is to achieve this by having read queries block until the data they read is majority committed, as opposed to reading from a majority committed snapshot. A core goal of this POC would be to get all (or most) of the change streams tests to pass when using this alternate implementation.&lt;/p&gt;</description>
                <environment></environment>
        <key id="606222">SERVER-37178</key>
            <summary>[POC] Allow change streams to work with &quot;blocking&quot; implementation of majority reads</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="william.schultz@mongodb.com">William Schultz</assignee>
                                    <reporter username="william.schultz@mongodb.com">William Schultz</reporter>
                        <labels>
                    </labels>
                <created>Mon, 17 Sep 2018 22:54:57 +0000</created>
                <updated>Wed, 10 Oct 2018 21:21:02 +0000</updated>
                            <resolved>Wed, 10 Oct 2018 20:48:26 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2029325" author="william.schultz" created="Wed, 10 Oct 2018 18:38:42 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=charlie.swanson&quot; class=&quot;user-hover&quot; rel=&quot;charlie.swanson&quot;&gt;charlie.swanson&lt;/a&gt; Yep, that is a test that doesn&apos;t play nice with the blocking majority reads implementation. It opens a change stream cursor, pauses replication on secondaries, inserts a document on the primary, and then waits to make sure the next returned batch is empty. By default, this just hangs when using blocking majority reads, since it will indefinitely wait on an optime to become committed that won&apos;t until replication is restarted. It is possible to make the test fail instead of hang if you specify a maxTimeMS on the initial aggregate command and the getMore. That is, it just returns an error instead of an empty batch. If there is a limited set of tests like this, we may consider writing analogous versions for the blocking majority reads implementation, or altering the test to support both implementations. I will think on this more.&lt;/p&gt;</comment>
                            <comment id="2029276" author="charlie.swanson" created="Wed, 10 Oct 2018 17:53:04 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.schultz&quot; class=&quot;user-hover&quot; rel=&quot;william.schultz&quot;&gt;william.schultz&lt;/a&gt; you point out a good gap in our coverage - we do have one test that I can find which specifically tests that the majority read concern is working: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/beda255a13b61570683968d6ac18a02f1d9bf765/jstests/noPassthrough/change_streams_require_majority_read_concern.js&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;change_streams_require_majority_read_concern.js&lt;/a&gt;. Does this test pass with your patch build? Or I just realize that perhaps it can no longer be run if you disable majority read concern support?&lt;/p&gt;</comment>
                            <comment id="2029261" author="william.schultz" created="Wed, 10 Oct 2018 17:36:39 +0000"  >&lt;p&gt;I think that this POC has successfully served its purpose to help inform the design of change streams with blocking majority reads. At this point, I think that more time spent working on POC code changes would be better served by just working on the full implementation. A diff of the POC changes is attached ( &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/198314/198314_blocking_majority_reads_poc.diff&quot; title=&quot;blocking_majority_reads_poc.diff attached to SERVER-37178&quot;&gt;blocking_majority_reads_poc.diff&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; , diffed against &lt;a href=&quot;https://github.com/mongodb/mongo/commit/860b392d9d3c006090a4c7fc3c6f3fa5460e5c5c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;860b39&lt;/a&gt;). The insights I feel were gained from this work are as follows:&lt;/p&gt;

&lt;p&gt;1. Gained a better understanding of existing change streams code and how to integrate blocking majority reads mechanism. One notable point is that we will likely need to do the majority waiting for getMore inside the command itself, and not in the generic command processing pipeline, since there&apos;s no good way to get access to the read concern associated with a cursor from within ServiceEntryPoint. Also, I learned that updateLookup for local mongod queries should automatically read from the same snapshot as the change event read, so it should automatically give majority semantics as long as we block after the command execution has finished.&lt;/p&gt;

&lt;p&gt;2. Acquired a better perspective on change streams test coverage, and also realized to be wary of existing coverage for testing the blocking majority reads behavior. It seems that there are many tests don&apos;t necessarily depend on the fact that change streams use majority read concern. That is, there are a lot of change streams tests that will read from a node whose lastApplied==lastCommitted. In this case, even if majority reads were disabled or weren&apos;t working correctly, it doesn&apos;t seem like these tests would notice it. For example, I &lt;em&gt;was&lt;/em&gt; able to get the basic &lt;tt&gt;change_streams&lt;/tt&gt;&#160;test suite to pass using the POC implementation, but this probably doesn&apos;t really tell us much since this only uses a single node replica set, there are no failovers, etc. When doing the full implementation, we should be careful to add some new change streams tests that explicitly test cases when change stream queries actually induce waiting on nodes, in addition to waiting for events against nodes that subsequently undergo roll back. I think that the Jepsen test for blocking majority reads should help improve the coverage in this area, since it induces many rollbacks throughout the test. I didn&apos;t think it was worth writing more extensive test coverage as a part of the POC, since it didn&apos;t feel valuable in terms of proving out the concept of the implementation. We may want to consider a change streams passthrough that kills nodes or induces rollbacks as well, if we think the time investment is worth it.&lt;/p&gt;</comment>
                            <comment id="2009541" author="william.schultz" created="Thu, 20 Sep 2018 17:53:58 +0000"  >&lt;p&gt;Note: It may make sense to implement this prototype on the 4.0 branch. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="616765">SERVER-37560</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="198314" name="blocking_majority_reads_poc.diff" size="15454" author="william.schultz@mongodb.com" created="Wed, 10 Oct 2018 18:41:12 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 10 Oct 2018 17:53:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 18 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1191</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 18 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>william.schultz@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu88vb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htyo5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2541">Repl 2018-10-08</customfieldvalue>
    <customfieldvalue id="2542">Repl 2018-10-22</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hu7v4n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>