<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:31:56 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-32988] Oplog application, foreground index builds pin an unbounded amount of data in WiredTiger</title>
                <link>https://jira.mongodb.org/browse/SERVER-32988</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Note, this only applies to the 3.7 development branch.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32188&quot; title=&quot;Have secondaries apply timestamps to commands&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32188&quot;&gt;&lt;del&gt;SERVER-32188&lt;/del&gt;&lt;/a&gt; &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f492aeddbf/src/mongo/db/repl/oplog.cpp#L1602-L1603&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;started to timestamp&lt;/a&gt; writes on secondaries from command oplog entries. It uses a &lt;tt&gt;`TimestampBlock`&lt;/tt&gt; to pass context on the OperationContext&apos;s &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f492aeddbf5e4dbd660d7aa297b0cdbe939c4906/src/mongo/db/storage/recovery_unit.h#L158-L169&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;RecoveryUnit&lt;/a&gt; which is applied when &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f492aeddbf/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L205-L209&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;transactions commit&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The index build uses a WriteUnitOfWork for &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f492aeddbf/src/mongo/db/catalog/index_create_impl.cpp#L348-L353&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;each document&lt;/a&gt; resulting from the collection scan. This write gets the &quot;commit timestamp&quot; on the recovery unit. Foreground index builds use the recovery unit that is in context from the &lt;tt&gt;TimestampBlock&lt;/tt&gt;. Background index builds use &lt;a href=&quot;https://github.com/mongodb/mongo/blob/f492aeddbf/src/mongo/db/index_builder.cpp#L85-L87&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;their own OperationContext&lt;/a&gt;. This causes background index builds to not timestamp their data writes.&lt;/p&gt;

&lt;p&gt;Moreover, foreground index builds block replication. When replication is not progressing, the &lt;tt&gt;oldest_timestamp&lt;/tt&gt; does not advance. If the &lt;tt&gt;oldest_timestamp&lt;/tt&gt; is not advancing, all of the data writes that are part of the index build stay pinned. This can unnecessarily activate lookaside.&lt;/p&gt;</description>
                <environment></environment>
        <key id="489286">SERVER-32988</key>
            <summary>Oplog application, foreground index builds pin an unbounded amount of data in WiredTiger</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="vesselina.ratcheva@mongodb.com">Vesselina Ratcheva</assignee>
                                    <reporter username="daniel.gottlieb@mongodb.com">Daniel Gottlieb</reporter>
                        <labels>
                            <label>rollback-non-functional</label>
                    </labels>
                <created>Tue, 30 Jan 2018 02:23:12 +0000</created>
                <updated>Fri, 27 Oct 2023 13:54:00 +0000</updated>
                            <resolved>Thu, 22 Mar 2018 13:56:35 +0000</resolved>
                                    <version>3.7.1</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1841837" author="daniel.gottlieb@10gen.com" created="Thu, 22 Mar 2018 13:56:35 +0000"  >&lt;p&gt;It turns out that bulk index builds on WiredTiger (whether the &quot;bulk&quot; option succeeds or not) are, subtly, done outside a begin/commit transaction. The constructor &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0fa9948c6155b34480943a1b8275210c60050497/src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp#L591&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;opens a cursor&lt;/a&gt; that is used to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0fa9948c6155b34480943a1b8275210c60050497/src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp#L666&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;perform all inserts&lt;/a&gt; into the index. These inserts are self-contained &quot;autocommit&quot; transactions that never have a timestamp applied.&lt;/p&gt;

&lt;p&gt;Even though the call is inside a &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0fa9948c6155b34480943a1b8275210c60050497/src/mongo/db/index/index_access_method.cpp#L499-L532&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;committed WUOW&lt;/a&gt;, a session &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0fa9948c6155b34480943a1b8275210c60050497/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L167-L169&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;had never become &quot;active&quot;&lt;/a&gt; (had a transaction start), and likewise &lt;a href=&quot;https://github.com/mongodb/mongo/blob/0fa9948c6155b34480943a1b8275210c60050497/src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp#L73-L75&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;commit is not called&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The side-effect of not doing this ticket is that other storage engine&apos;s that obey the timestamping contract (of which there are none...that don&apos;t extend from WT itself) may pin index builds in memory. However, completing this ticket in a way that includes proof that it was done correctly would require changes to WTRecoveryUnits and/or WT index builds.&lt;/p&gt;</comment>
                            <comment id="1793182" author="ian@10gen.com" created="Fri, 2 Feb 2018 15:12:06 +0000"  >&lt;p&gt;just bumping this to repl team to make sure repl team sees it.  will also move kyle&apos;s work over to repl team asap.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 2 Feb 2018 15:12:06 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 46 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-842</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 46 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>daniel.gottlieb@mongodb.com</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>vesselina.ratcheva@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htp2m7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht956n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2146">Repl 2018-02-26</customfieldvalue>
    <customfieldvalue id="2185">Repl 2018-03-12</customfieldvalue>
    <customfieldvalue id="2186">Repl 2018-03-26</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htooqv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>