<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:25:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30873] Investigate where the oplog truncation happens during replication recovery</title>
                <link>https://jira.mongodb.org/browse/SERVER-30873</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/replication_recovery.h#L85&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;comments&lt;/a&gt; for ReplicationRecovery::_truncateOplogTo indicate that it should truncate entries after and including &apos;truncateTimestamp&apos;.&lt;/p&gt;

&lt;p&gt;The &lt;a href=&quot;https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/replication_recovery.cpp#L272&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;implementation&lt;/a&gt;, however, finds the first oplog entry &lt;b&gt;less&lt;/b&gt; than the &apos;truncateTimestamp&apos; then tells the storage engine to truncate the oplog starting at that entry, and passes true for the &apos;inclusive&apos; flag.  This means it will actually truncate the entry one before the &apos;truncateTimestamp&apos; passed it.&lt;/p&gt;

&lt;p&gt;We need to audit exactly how and when the oplogTruncateAfterPoint is set and make sure we&apos;re truncating at the right points.&lt;/p&gt;</description>
                <environment></environment>
        <key id="421418">SERVER-30873</key>
            <summary>Investigate where the oplog truncation happens during replication recovery</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="judah.schvimer@mongodb.com">Judah Schvimer</assignee>
                                    <reporter username="spencer@mongodb.com">Spencer Brody</reporter>
                        <labels>
                    </labels>
                <created>Tue, 29 Aug 2017 15:49:55 +0000</created>
                <updated>Fri, 27 Oct 2023 13:54:16 +0000</updated>
                            <resolved>Tue, 12 Sep 2017 18:16:25 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="1670956" author="daniel.gottlieb@10gen.com" created="Tue, 12 Sep 2017 18:40:32 +0000"  >&lt;p&gt;That makes sense. What I was doing with integrating rollback via recovery was observing some problem in this area, but perhaps I misdiagnosed it. I&apos;m with Eric; let&apos;s close this for now and I&apos;m sure if there&apos;s a real problem it will show up again when the project is revisited.&lt;/p&gt;</comment>
                            <comment id="1670815" author="milkie" created="Tue, 12 Sep 2017 18:13:40 +0000"  >&lt;p&gt;There might still be some bug here, as I believe the problem involves the intersection of the stable timestamp and the truncation point.  The unit test is just hard coding some value for the truncate-after-point, so while the logic is confirmed correct for truncation point, there may still be a problem at a higher layer.  I think we can resolve this particular ticket, though, and reopen a new ticket after we experience the problem again.&lt;/p&gt;</comment>
                            <comment id="1670764" author="spencer" created="Tue, 12 Sep 2017 17:53:13 +0000"  >&lt;p&gt;Makes sense to me.  This came up because of an issue &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt; was having in his testing.  Dan it seems like there&apos;s no bug here after all.  Do your findings disagree with that?&lt;/p&gt;

&lt;p&gt;FYI &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1670481" author="judah.schvimer" created="Tue, 12 Sep 2017 15:35:14 +0000"  >&lt;p&gt;I believe this is actually fine and the code is doing what it&apos;s supposed to be doing as evidenced by this unittest: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bddfee0d34513631424363645139f8eb4acbbc4c/src/mongo/db/repl/replication_recovery_test.cpp#L263-L279&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/bddfee0d34513631424363645139f8eb4acbbc4c/src/mongo/db/repl/replication_recovery_test.cpp#L263-L279&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The code is a bit confusing, but we truncate the oplog at &lt;a href=&quot;https://github.com/mongodb/mongo/blob/bddfee0d34513631424363645139f8eb4acbbc4c/src/mongo/db/repl/replication_recovery.cpp#L282&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;`oldestIDToDelete`&lt;/a&gt;, not at the current RecordID we&apos;re looking at. We search backwards through the oplog, checking if we&apos;re past the `truncateTimestamp` at each point. If we are not past the `truncateTimestamp` (greater than or equal to it), then we set `oldestIDToDelete` equal to the current oplog entry we are looking at. If we are past the `truncateTimestamp`, we do NOT set the `oldestIDToDelete` again, and pass the previous `oldestIDToDelete` into `cappedTruncateAfter`, which must be greater than or equal to `truncateTimestamp`.&lt;/p&gt;

&lt;p&gt;Example: We have an oplog &lt;span class=&quot;error&quot;&gt;&amp;#91;1,2,3&amp;#93;&lt;/span&gt; and we want to truncate at 2 yielding &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt; afterwards. We begin looking at 3. 3 is not less than 2, so we set `oldestIDToDelete` to 3. We then look at 2. 2 is not less than 2 so we set `oldestIDToDelete` to 2. We then look at 1. 1 is less than 2, so we truncate at `oldestIDToDelete`(inclusive) which is 2. This leads to our desired oplog of &lt;span class=&quot;error&quot;&gt;&amp;#91;1&amp;#93;&lt;/span&gt;. &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt;, do you agree with this reading and does the unittest suffice?&lt;/p&gt;</comment>
                            <comment id="1659630" author="spencer" created="Tue, 29 Aug 2017 15:50:36 +0000"  >&lt;p&gt;This also appears to be an issue on older versions as well.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="14340"><![CDATA[v3.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 Sep 2017 15:35:14 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 22 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-842</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 22 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htdu1j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht1tbr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1893">Repl 2017-10-02</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htdg4f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>