<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:53:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-20328] Allow secondary reads while applying oplog entries</title>
                <link>https://jira.mongodb.org/browse/SERVER-20328</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently replication applies batches of oplog entries holding a lock to ensure that no reads can consume data which is not in the same causal order as the primary. &lt;/p&gt;

&lt;p&gt;Instead of locking and blocking readers we can instead use a snapshot for reads from the last consistent replication state and essentially hide all writes until we reach a new consistent state. In addition to detaching replication writes from affecting readers (both users and other replicas) this also allows storage engines to optimize writing of replicated data to improve performance and reduce IO, since all replicated data is transient and disposable until committed.&lt;/p&gt;</description>
                <environment></environment>
        <key id="228962">SERVER-20328</key>
            <summary>Allow secondary reads while applying oplog entries</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="louis.williams@mongodb.com">Louis Williams</assignee>
                                    <reporter username="scotthernandez">Scott Hernandez</reporter>
                        <labels>
                    </labels>
                <created>Wed, 9 Sep 2015 15:01:41 +0000</created>
                <updated>Mon, 15 Nov 2021 15:11:42 +0000</updated>
                            <resolved>Fri, 13 Apr 2018 16:11:35 +0000</resolved>
                                                                    <component>Replication</component>
                    <component>Storage</component>
                                        <votes>22</votes>
                                    <watches>36</watches>
                                                                                                                <comments>
                            <comment id="1863764" author="louis.williams" created="Fri, 13 Apr 2018 16:11:35 +0000"  >&lt;p&gt;Completed by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34192&quot; title=&quot;Secondary reads during batch applications&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34192&quot;&gt;&lt;del&gt;SERVER-34192&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1572934" author="zyd_com@126.com" created="Wed, 17 May 2017 02:28:45 +0000"  >&lt;p&gt;@deyukong @Eric&lt;/p&gt;

&lt;p&gt;We also encountered this issue, as mentioned in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24661&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;SERVER-24661&lt;/a&gt; , and it&apos;s linked to this &lt;b&gt;&lt;font color=&quot;red&quot;&gt;Read From Snapshots&lt;/font&gt;&lt;/b&gt; issue, which making no progress all the time.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/WT-3181&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;WT-3181&lt;/a&gt; is really helpful for solve this problem. I want to know the release plan,  will it be included in mongodb-3.6?&lt;/p&gt;</comment>
                            <comment id="1572394" author="wolf_kdy" created="Tue, 16 May 2017 15:56:11 +0000"  >&lt;p&gt;Yes, you&apos;re right.&lt;br/&gt;
If customers use readMajority and writeMajority, the problem will not happen. But I think more consideration should be taken here since customers may not be willing to use read/write Majority for some reasons.&lt;br/&gt;
A monitor to watch disk usage can also solve this problem and actually it is a tradeoff here.&lt;br/&gt;
The work of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3181&quot; title=&quot;Add support for MongoDB timestamps&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3181&quot;&gt;&lt;del&gt;WT-3181&lt;/del&gt;&lt;/a&gt; is great and worth appreciated.&lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/thumbs_up.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="1572373" author="milkie" created="Tue, 16 May 2017 15:40:04 +0000"  >&lt;p&gt;If a majority of secondaries are stale, you are already in dangerous territory as writes are not propagating to a majority of nodes (and writes done with a write concern of w:majority will time out).  For such a situation, it will be possible for the secondary nodes to accumulate snapshot data until cache memory is full and then start spilling the snapshot data to disk.  The solution is to use w:majority writes, or set up monitoring to avoid running out of disk space.&lt;/p&gt;</comment>
                            <comment id="1572354" author="wolf_kdy" created="Tue, 16 May 2017 15:27:55 +0000"  >&lt;p&gt;quoted:&lt;br/&gt;
&lt;em&gt;The data enabling these snapshots will be cleaned up as the commit point moves forward&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In the current readMajority implementation, only 1000 snapshots will be held. If you try to keep snapshots since the committed-point in the snapshot-manager,  there is a chance that a secondary will accumulate so much data on disk once majority of secondaries are stale but one is not.&lt;br/&gt;
Commited-point is unpredictable, I would rather do named snapshots manually to avoid the accumulation of an alive secondary&lt;/p&gt;</comment>
                            <comment id="1572319" author="milkie" created="Tue, 16 May 2017 15:12:52 +0000"  >&lt;p&gt;Once the timestamp project is fully completed, you will be able to do a point-in-time read on a node at any optime between now and the majority commit point with snapshot isolation.  In essence, we will be making snapshots automatically for every op in the oplog.  The data enabling these snapshots will be cleaned up as the commit point moves forward; in most cases it will not be possible to begin a read at a point in time prior to the current commit point (but in-flight reads will pin their snapshot until they are finished).&lt;/p&gt;</comment>
                            <comment id="1572304" author="wolf_kdy" created="Tue, 16 May 2017 15:08:10 +0000"  >&lt;p&gt;Hi, @Eric Mikie&lt;br/&gt;
That&apos;s really a great feature, with this feature, each oplog will be exactly mapped to some version of a key-value pair in wt. But I think it will be hard to deal with meta-data changing, Such as renameCollection or something similar, I&apos;m not quite sure if renameCollection will change the wt-related idents or not, I will review this point later.&lt;br/&gt;
You said, quoted below&lt;br/&gt;
&quot;&lt;br/&gt;
&lt;em&gt;With this new feature, we&apos;ll be able to allow reads on secondaries for both read concern level majority and read concern level local without blocking while a batch is being applied&lt;/em&gt;&lt;br/&gt;
&quot;&lt;br/&gt;
Certainly it will be more convient to do this with your new feature. But I think it is not a  sine qua non. It may not be convient to do and release named-snapshot before and after the batchApplyOplog, but it is able to fulfill the task. &lt;br/&gt;
Or possibily, &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3181&quot; title=&quot;Add support for MongoDB timestamps&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3181&quot;&gt;&lt;del&gt;WT-3181&lt;/del&gt;&lt;/a&gt; is one milestone of the roadmap to do snapshot-read on secondaries ?&lt;/p&gt;</comment>
                            <comment id="1572178" author="milkie" created="Tue, 16 May 2017 13:49:26 +0000"  >&lt;p&gt;Hi Deyu Kong,&lt;br/&gt;
We are currently working on a project &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3181&quot; title=&quot;Add support for MongoDB timestamps&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3181&quot;&gt;&lt;del&gt;WT-3181&lt;/del&gt;&lt;/a&gt; to label all writes with timestamps, so that point-in-time reads will be possible without needing to create named snapshots ahead of time.  With this new feature, we&apos;ll be able to allow reads on secondaries for both read concern level majority and read concern level local without blocking while a batch is being applied.&lt;/p&gt;</comment>
                            <comment id="1571202" author="wolf_kdy" created="Mon, 15 May 2017 15:10:56 +0000"  >&lt;p&gt;I&apos;m one from tencent-cloud-mongodb team, in our real-world case, there are many many customs troubled by this problem.&lt;br/&gt;
Let&apos;s talk about one of our customer, the most famous shared-bicycle service provider in china(the name is skipped here by security)&lt;br/&gt;
there are two period of time during a day when people go to work in the morning or get off work in the afternoon. During this&lt;br/&gt;
period, tps may be four times as high as the normal time. Although we route reads into the secondaries and use primary only to serve writes. We still find slow queries got extremely high, maybe hundards of times as the normal time.&lt;br/&gt;
After some time of debug, we finally find the PBWM lock is the main reason. When primary writes heavily, secondaries will take&lt;br/&gt;
lots of time acquiring global intent lock(in our case, all data are cached in memory, primary write/update may be 10000--&lt;del&gt;20000/s at peak, each slave costs 50&lt;/del&gt;-200ms mostly for applying oplogs, we got the time of applying oplogs because we found it and we add logs around), the time to apply oplog is very unstable, sometimes less than 10ms, sometimes greater than 500ms.&lt;br/&gt;
The unstable performance makes the time of reading slave unpredictable, even a simple key-value read can cost very long time. it is untolerable to a database. We think the most important thing of a database is the stablilization, then the performance, the last important is the variaty of features.&lt;br/&gt;
We are hoping the offical mongodb guys to focus on this problem and list out the roadmap to solve it.&lt;br/&gt;
As one of the community, I have some ideas to share here, ideas about how to solve this problem.&lt;br/&gt;
1) Primary must not read from snapshot, because the read/write consistency may be destroyed to do so.&lt;br/&gt;
2) Secondaries do a snapshot before or after BatchApplyOplogs( I have not tested wt&apos;s performance of snapshot, but I know each read or write of mongorocks uses a snapshot, if frequent snapshots is a problem, tradeoff between frequency and latency&lt;br/&gt;
should be considered)&lt;br/&gt;
3) Reads can be classified into two kinds, reading from the process itself and reading from the clients(other mongod/s included)&lt;br/&gt;
4) Add some flag to recoverUnit to read from snapshot, just like what you did to implement readMajority, track from the beginning of each reads from client, set this flag and they will read from the lastest snapshot&lt;br/&gt;
5) The modules reads inside mongod process as far as I know is:&lt;br/&gt;
5.1) get the latest oplog in cluster-heartbeat for raft replication. it should also read from the latest snapshot, otherwise, an oplog generated from the partial-oplogBatchApply may be observed.&lt;br/&gt;
5.2) somewhere else ?&lt;br/&gt;
6) I&apos;ve sketched the main framework of reading from snapshot in secondaries, which can be found here &lt;a href=&quot;https://github.com/wolfkdy/mongo/tree/snapshot_read&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;snapshot_read&lt;/a&gt;, which mainly fulfilles the points I listed above.&lt;/p&gt;

&lt;p&gt;There must be some points that I&apos;ve missed, and needs further discussions. I hope the official mongo guys can participate in.&lt;br/&gt;
Btw, I got some doubts when sketching out the framework, which is listed below, would someone be so kind to answer it?&lt;br/&gt;
Thanks &lt;a href=&quot;https://groups.google.com/forum/#!topic/mongodb-dev/uVd6or43TO8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/#!topic/mongodb-dev/uVd6or43TO8&lt;/a&gt;&lt;/p&gt;


</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="518833">SERVER-34192</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="243424">SERVER-21858</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="295059">SERVER-24661</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="288090">WT-2649</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="243461">SERVER-21862</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="302600">SERVER-25168</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="37494">SERVER-5729</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="48450">SERVER-6883</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="382695">SERVER-29123</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="439084">SERVER-31359</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000Y6rMkIAJ, 500A000000XzMBZIA3, 500A000000YfK8jIAF, 500A000000YfNCeIAN, 500A000000YfTV9IAN]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 15 May 2017 15:10:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 43 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-644</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 43 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>wolf_kdy</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
            <customfieldvalue>zyd_com@126.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkv8v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9daf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs1lmv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>