<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:50:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-82734] Improve capped collection write performance.</title>
                <link>https://jira.mongodb.org/browse/SERVER-82734</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While investigating&#160;HELP-51123, I found some potential areas where we can improve the capped collection performance.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Delete code path&lt;/b&gt;&lt;br/&gt;
&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&lt;/p&gt;

&lt;p&gt;1) Don&apos;t serialize capped deletes on secondaries.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Oplog applier currently serializes &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/repl/oplog_applier_utils.cpp#L122-L143&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;both capped inserts/updates and deletes &lt;/a&gt;. While serializing inserts/updates is necessary for maintaining consistent natural ordering across the replica set, serializing deletes is unnecessary.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;2) Do batched deletion (like, PM-2227) before performing vectored inserts.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Currently, we perform batched deletes (doing multiple deletes in a single storage transaction). However, we &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/catalog/capped_collection_maintenance.cpp#L178-L207&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;reserve oplog slots for capped deletes one by one&lt;/a&gt;. Given the existing issue with vectored inserts related to timestamp interleaving, this approach could additionally slow down vectored inserts for capped collections, especially since we&apos;re waiting for capped deletes to achieve majority replicated.&lt;/li&gt;
	&lt;li&gt;In addition, we&apos;re checking the capped collection size and performing deletes after each capped inserts, which may not be the most efficient process.&lt;/li&gt;
	&lt;li&gt;As an optimization, we should consider reserving oplog slots in bulk and performing batched operations &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/ops/write_ops_exec.cpp#L1127&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;before executing vector inserts&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;3) Another optimization to consider is using truncate instead of delete calls for substantial numbers of deletions. In 4.4, we used &lt;a href=&quot;https://github.com/10gen/mongo/blob/9f67f47c4d095943e5e3e05e46fc5c907aad44ac/src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp#L115-L125&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;truncate&lt;/a&gt; when the number of documents to be deleted was &amp;gt; 3. Since, we were doing unreplicated implicit deletes in 4.4 and older version, this was simpler to handle corresponding index entry deletes. Since the start of 5.0, we&apos;ve been using replicated delete, that might impose some challenges with index deletes. But, this may be worth exploring in PM-2983.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Insert code path&lt;/b&gt;&lt;br/&gt;
&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt;&lt;br/&gt;
1) Enable group inserts on secondaries.&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Given that we do replicated deletes starting from 5.0, we can enable &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/repl/insert_group.cpp#L92-L94&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;group inserts&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;2) Enable batch inserts on primary (see &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/ops/write_ops_exec.cpp#L644&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt; and &lt;a href=&quot;https://github.com/10gen/mongo/blob/463395b4aafd5c0e85a1dbcc28b5c5b275469886/src/mongo/db/catalog/collection_write_path.cpp#L215&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;)&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;This will be safe if we also do &quot;Delete code path&quot; optimization #2.&lt;/li&gt;
&lt;/ul&gt;
</description>
                <environment></environment>
        <key id="2489589">SERVER-82734</key>
            <summary>Improve capped collection write performance.</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10033" iconUrl="https://jira.mongodb.org/images/icons/statuses/information.png" description="Status for tickets that need to be escalated and unblocked on our team.">Blocked</status>
                    <statusCategory id="4" key="indeterminate" colorName="inprogress"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-execution">Backlog - Storage Execution Team</assignee>
                                    <reporter username="suganthi.mani@mongodb.com">Suganthi Mani</reporter>
                        <labels>
                    </labels>
                <created>Thu, 2 Nov 2023 20:39:35 +0000</created>
                <updated>Thu, 16 Nov 2023 16:45:02 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="5885781" author="steven.vannelli" created="Thu, 16 Nov 2023 16:44:31 +0000"  >&lt;p&gt;Backlogging this until we finish PM-2983: Globally Unique, Replicated Record Ids to see what comes out of that project. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25136"><![CDATA[Storage Execution]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 16 Nov 2023 16:44:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<a href='https://jira.mongodb.org/browse/PM-2983'>PM-2983</a>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>steven.vannelli@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-execution</customfieldvalue>
            <customfieldvalue>steven.vannelli@mongodb.com</customfieldvalue>
            <customfieldvalue>suganthi.mani@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2ynyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2giak:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2ya3z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>