<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:00:36 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-64547] Improve batched deletion performance on secondaries</title>
                <link>https://jira.mongodb.org/browse/SERVER-64547</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently secondaries do not batch deletions as the primary does.The secondary has been observed to become a bottleneck in some cases. Explore potential improvements to bring the secondaries on par with the primary.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2002761">SERVER-64547</key>
            <summary>Improve batched deletion performance on secondaries</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13202">Works as Designed</resolution>
                                        <assignee username="josef.ahmad@mongodb.com">Josef Ahmad</assignee>
                                    <reporter username="josef.ahmad@mongodb.com">Josef Ahmad</reporter>
                        <labels>
                            <label>PM-2227-M3</label>
                    </labels>
                <created>Wed, 16 Mar 2022 09:12:39 +0000</created>
                <updated>Fri, 27 Oct 2023 13:52:04 +0000</updated>
                            <resolved>Mon, 2 May 2022 07:24:37 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4519282" author="josef.ahmad" created="Mon, 2 May 2022 07:24:20 +0000"  >&lt;p&gt;Resolving this ticket as working as designed. Per my initial comment, in my benchmarks with 1kB-sized documents, the batched deleter shows 3x the throughput of the doc-by-doc deleter, and lower impact (in terms of latency of concurrent w:majority writes) than running 3 doc-by-doc mass deletions in parallel. There is no evidence of a performance bottleneck that is specific to a secondary member.&lt;/p&gt;</comment>
                            <comment id="4519278" author="josef.ahmad" created="Mon, 2 May 2022 07:17:52 +0000"  >&lt;p&gt;Turns out that the behaviour I was observing is not specific to secondaries, it&apos;s just more apparent on secondaries because of replication rules. Explanation below.&lt;br/&gt;
&#160;&lt;br/&gt;
On the primary, the concurrent inserts can interleave freely with the mass document removal, so if storage is bottlenecked on read path (e.g. during a checkpoint per my previous comment), the remove ops would be bound by storage although the inserts can progress with no impact.&lt;br/&gt;
On a secondary however, the same storage-bound scenario does impact the inserts: writes must replicate in the same order (across batches) as they executed on the primary. So if the system is read-storage bound, the inserts must wait on the outstanding deletes, and this in turn impacts the rate at which inserts are replicated. This is why the latency of majority write concern inserts increases during checkpoints on secondaries.&lt;br/&gt;
&#160;&lt;br/&gt;
This behaviour holds up with larger documents (16kB) and in the presence of secondary indexes. I haven&apos;t seen evidence of the IDHACK fetch stage being problematic.&lt;/p&gt;</comment>
                            <comment id="4509632" author="josef.ahmad" created="Wed, 27 Apr 2022 13:41:45 +0000"  >&lt;p&gt;With 1kB sized documents, the batch deleter shows ~3x the throughput of the doc-by-doc deleter. And it&apos;s also faster than 3 doc-by-doc mass deletions occurring in parallel.&lt;br/&gt;
&#160;&lt;br/&gt;
In terms of latencies of concurrent w:majority inserts:&lt;br/&gt;
&#160;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;Concurrent w:majority inserts&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;p50 (ms)&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;p99 (ms)&lt;/b&gt;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&lt;b&gt;max (ms)&lt;/b&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;During doc-by-doc mass deletion&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;5.6&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;7.6&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;During 3x doc-by-doc mass deletion&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;15.7&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;42.2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2603&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;During batched deletion&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;12&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;33.7&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2103&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;&#160;&lt;br/&gt;
The tail latency is due to secondary replication. This seems to be a consequence of the primary deleting documents at a higher rate than before, and it&apos;s still lower than the tail latency of running multiple doc-by-doc deletions in parallel.&lt;br/&gt;
&#160;&lt;br/&gt;
The chart below shows one such latency spike between A and B. Off-CPU analysis (blue metrics in the &quot;Thread profile&quot; section) reveals the latency is due to a checkpoint&apos;s fdatasync on the secondary which saturates the disk and impacts fetching a document for deletion.&lt;br/&gt;
&#160;&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/375934/375934_image-2022-04-27-15-40-54-912.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;br/&gt;
&#160;&lt;br/&gt;
It would be interesting to determine why this behaviour is only observed on secondaries, although I&apos;m not sure that is in scope for ticket. Relevant questions would be whether the checkpoints on secondaries are less efficient than on the primary, and whether the document fetches when replicating writes (IDHACK) are less efficient than the ranged fetches that occur on the primary.&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2040690">SERVER-66255</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="375934" name="image-2022-04-27-15-40-54-912.png" size="309344" author="josef.ahmad@mongodb.com" created="Wed, 27 Apr 2022 13:40:59 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 40 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2227</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 40 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>josef.ahmad@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0nggn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr2mac:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5812">Execution Team 2022-03-21</customfieldvalue>
    <customfieldvalue id="5814">Execution Team 2022-04-18</customfieldvalue>
    <customfieldvalue id="5979">Execution Team 2022-05-02</customfieldvalue>
    <customfieldvalue id="5980">Execution Team 2022-05-16</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0n2lz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>