<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:38:26 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-34977] subtract capped deletes from fastcount during replication recovery</title>
                <link>https://jira.mongodb.org/browse/SERVER-34977</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The following &quot;algebra&quot; should explain what&apos;s happening currently. One question is, do capped deletes get undone when we recover to a timestamp. Assuming they do get undone, we have the following.&lt;/p&gt;

&lt;p&gt;&#160;&lt;br/&gt;
itcount and fastcount at stable timestamp = A&lt;br/&gt;
Operations between stable timestamp and common point = B&#160;&lt;br/&gt;
capped deletes between stable timestamp and common point = Cd1&lt;/p&gt;

&lt;p&gt;Operations between common point and top of oplog = Diff&lt;br/&gt;
capped deletes between common point and top of oplog = Cd2&lt;br/&gt;
itcount and fastcount when rollback begins = A + B + Diff - Cd1 - Cd2&lt;br/&gt;
&#160;&lt;br/&gt;
&#8212; recover to stable timestamp ----&lt;br/&gt;
&#160;&lt;br/&gt;
itcount = A&lt;br/&gt;
fastcount = A&#160;+ B&#160;+ Diff - Cd1 -&#160;Cd2&lt;br/&gt;
&#160;&lt;br/&gt;
&#8212; replication recovery &#8212;&lt;br/&gt;
&#160;&lt;br/&gt;
Operations during replication recovery = B&lt;br/&gt;
capped deletes during replication recovery = Cd3&lt;br/&gt;
itcount after replication recovery =&#160;A + B - Cd3&lt;br/&gt;
fast count = ??? (depending on collections marked for size adjustment)&lt;br/&gt;
&#160;&lt;br/&gt;
&#8212; reset counts after rollback by Diff &#8212;&lt;br/&gt;
&#160;&lt;br/&gt;
itcount =&#160;A + B - Cd3&lt;br/&gt;
fastcount =&#160;A + B + Diff - Cd1 - Cd2 - Diff =&#160;A + B - Cd1 - Cd2&lt;br/&gt;
&#160;&lt;br/&gt;
This is a problem since we have no idea of Cd1 and Cd2. If, however, capped deletes are not undone (i.e. they&apos;re not timestamped), then itcount after recover to stable timestamp is A - Cd1 - Cd2, and the itcount after replication recovery = A&#160;+ B - Cd1 - Cd2 - Cd3.&lt;br/&gt;
In that case we can just subtract Cd3 to be correct. It should be safe for capped deletes to not recover to a timestamp since users expect it to be safe for them to get aged out anyways.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;We can either keep track of the capped deletions and subtract them out, or turn off capped deletion during replication recovery and do it all at once at the end.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I think this applies to both rollback and replication recovery during startup, but there may be a reason it doesn&apos;t happen at startup.&lt;/p&gt;</description>
                <environment></environment>
        <key id="543453">SERVER-34977</key>
            <summary>subtract capped deletes from fastcount during replication recovery</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="judah.schvimer@mongodb.com">Judah Schvimer</assignee>
                                    <reporter username="judah.schvimer@mongodb.com">Judah Schvimer</reporter>
                        <labels>
                    </labels>
                <created>Mon, 14 May 2018 13:46:48 +0000</created>
                <updated>Sun, 29 Oct 2023 22:31:50 +0000</updated>
                            <resolved>Tue, 12 Jun 2018 18:18:09 +0000</resolved>
                                                    <fixVersion>4.0.0-rc6</fixVersion>
                    <fixVersion>4.1.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>6</watches>
                                                                                                                <comments>
                            <comment id="1918699" author="xgen-internal-githook" created="Tue, 12 Jun 2018 18:27:32 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;judahschvimer&apos;, &apos;name&apos;: &apos;Judah Schvimer&apos;, &apos;email&apos;: &apos;judah@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34977&quot; title=&quot;subtract capped deletes from fastcount during replication recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34977&quot;&gt;&lt;del&gt;SERVER-34977&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35435&quot; title=&quot;Renaming during replication recovery incorrectly allows size adjustments&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35435&quot;&gt;&lt;del&gt;SERVER-35435&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34976&quot; title=&quot;clear the &amp;quot;needing size adjustment&amp;quot; set at the beginning of replication rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34976&quot;&gt;&lt;del&gt;SERVER-34976&lt;/del&gt;&lt;/a&gt; Fix fastcounts for RTT, including capped collections.&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 8b698cac2d19f0fec502db10501e7059a10d2897)&lt;br/&gt;
Branch: v4.0&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/0fd5d4eb2e61bbef14f6e55c8e5f9619e807260b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/0fd5d4eb2e61bbef14f6e55c8e5f9619e807260b&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1918675" author="xgen-internal-githook" created="Tue, 12 Jun 2018 18:17:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;judahschvimer&apos;, &apos;name&apos;: &apos;Judah Schvimer&apos;, &apos;email&apos;: &apos;judah@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34977&quot; title=&quot;subtract capped deletes from fastcount during replication recovery&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34977&quot;&gt;&lt;del&gt;SERVER-34977&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-35435&quot; title=&quot;Renaming during replication recovery incorrectly allows size adjustments&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-35435&quot;&gt;&lt;del&gt;SERVER-35435&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-34976&quot; title=&quot;clear the &amp;quot;needing size adjustment&amp;quot; set at the beginning of replication rollback&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-34976&quot;&gt;&lt;del&gt;SERVER-34976&lt;/del&gt;&lt;/a&gt; Fix fastcounts for RTT, including capped collections.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8b698cac2d19f0fec502db10501e7059a10d2897&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/8b698cac2d19f0fec502db10501e7059a10d2897&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1911648" author="judah.schvimer" created="Tue, 5 Jun 2018 19:30:47 +0000"  >&lt;p&gt;After talking to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.gottlieb&quot; class=&quot;user-hover&quot; rel=&quot;daniel.gottlieb&quot;&gt;daniel.gottlieb&lt;/a&gt;, we&apos;ve realized that this analysis missed a key point, and that is that replication recovery should NEVER do a capped deletion since it is replaying operations we already played forward, and the documents that were deleted do not return at the stable timestamp/checkpoint after rollback/clean shutdown. There are two cases. &lt;/p&gt;

&lt;p&gt;The first is where we only truncate documents that were inserted behind the stable timestamp. In that case, when we jump back to the stable timestamp and insert the same documents into the collection again, the documents we should be deleting were already deleted the first time we played those operations. In this case, during replication recovery the data size will be greater than 0 and the collection will not be marked for size adjustment. When the record store checks to see if it should delete records on inserts/updates, it will use the pre-rollback size as the starting point and think that the size is actually bigger than it is. In that case it would try to do capped deletes like we&apos;re seeing, even though we shouldn&apos;t expect it to. &lt;/p&gt;

&lt;p&gt;The other case is where we truncate documents that were inserted during replication recovery. In that case we are guaranteed that the record store will be empty at the stable timestamp and we will mark the collection for size adjustment &lt;a href=&quot;https://github.com/mongodb/mongo/blob/7507298b1f05d46fdaf71e9f43f9a0b4ba577326/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L732-L742&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. The size then is corrected to 0 appropriately and we will allow capped deletions, which will be accounted for correctly in the count. &lt;/p&gt;

&lt;p&gt;We currently do not adjust the data size of a collection after a rollback. Thus the effective size of a capped collection may not actually be the real size of the capped collection. This is a bug we are accepting. A validate will fix this.&lt;/p&gt;</comment>
                            <comment id="1894640" author="judah.schvimer" created="Thu, 17 May 2018 20:52:42 +0000"  >&lt;p&gt;Please make sure to remove&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/commit/8b02420bc4eb76344b79b4bf45dbcf0cc6bed584&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this block&lt;/a&gt;&#160;in this ticket to ensure the fastcounts are correct.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="554805">SERVER-35435</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="555530">SERVER-35483</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="545712">SERVER-35052</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="543446">SERVER-34976</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1540658">SERVER-52833</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="554780">SERVER-35431</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15640"><![CDATA[v4.0]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 Jun 2018 18:17:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 35 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 35 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>65.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>judah.schvimer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htxza7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htp19z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2296">Repl 2018-06-04</customfieldvalue>
    <customfieldvalue id="2297">Repl 2018-06-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htxljj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>