<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:38:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-56194] Make TTL deletes fair</title>
                <link>https://jira.mongodb.org/browse/SERVER-56194</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The single-threaded TTL Monitor can get &quot;stuck&quot; deleting large ranges of documents on specific collections or databases. &lt;/p&gt;

&lt;p&gt;This prevents the TTL monitor from performing deletes on higher-priority collections such as config.system.sessions.&lt;/p&gt;

&lt;p&gt;We should consider imposing configurable per-database and per-collection document deletion limits. In addition, we will need to consider significantly lowering the default TTL pass interval (from 60 seconds) to ensure we make progress.&lt;/p&gt;

&lt;p&gt;Some hypothetical limits would be:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Maximum 10000 documents per database&lt;/li&gt;
	&lt;li&gt;Maximum 1000 documents per collection&lt;/li&gt;
	&lt;li&gt;Run TTL monitor every 5 seconds.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Also consider prioritizing important collections like config.system.sessions.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1683953">SERVER-56194</key>
            <summary>Make TTL deletes fair</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="haley.connelly@mongodb.com">Haley Connelly</assignee>
                                    <reporter username="louis.williams@mongodb.com">Louis Williams</reporter>
                        <labels>
                            <label>PM-2227-M3</label>
                    </labels>
                <created>Tue, 20 Apr 2021 14:41:07 +0000</created>
                <updated>Thu, 16 Nov 2023 22:52:33 +0000</updated>
                            <resolved>Mon, 30 May 2022 08:36:48 +0000</resolved>
                                                    <fixVersion>6.1.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>20</watches>
                                                                                                                <comments>
                            <comment id="4578390" author="xgen-internal-githook" created="Fri, 27 May 2022 17:07:56 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Haley Connelly&apos;, &apos;email&apos;: &apos;haley.connelly@mongodb.com&apos;, &apos;username&apos;: &apos;haleyConnelly&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-56194&quot; title=&quot;Make TTL deletes fair&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-56194&quot;&gt;&lt;del&gt;SERVER-56194&lt;/del&gt;&lt;/a&gt; Make TTL deletes fair&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1e84419513046d4f5755f06bfeebaf8ec442583e&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1e84419513046d4f5755f06bfeebaf8ec442583e&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4507259" author="haley.connelly" created="Tue, 26 Apr 2022 16:47:38 +0000"  >&lt;p&gt;After chatting with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=louis.williams%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;louis.williams@mongodb.com&quot;&gt;louis.williams@mongodb.com&lt;/a&gt;, for simplicity, we decided to bound the deletes per collection and not enforce fairness per database.&lt;/p&gt;

&lt;p&gt;Things get complicated when a cache of collection UUIDs turns into a perDB structure that needs to enforce some sort of order / fairness and keep track of which collection the previous pass left off on if a dbLimit was reached. Preventing collection starvation, and accounting for new collections and dropped collections, for each database, would increase complexity.&#160;&lt;/p&gt;</comment>
                            <comment id="4027415" author="haley.connelly" created="Fri, 27 Aug 2021 20:00:58 +0000"  >&lt;p&gt;Currently, there is no easy way to limit the number of documents/ amount of work done while deleting documents via an index scan. While this could be pushed to the query level, we determined fair TTL deletions will likely benefit from the work done to improve truncate for efficient range deletion.&lt;/p&gt;

&lt;p&gt;Deferring this ticket until TTL deletions can utilize a more efficient truncate for range deletions.&lt;/p&gt;</comment>
                            <comment id="3728199" author="bruce.lucas@10gen.com" created="Tue, 20 Apr 2021 17:01:32 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=milkie&quot; class=&quot;user-hover&quot; rel=&quot;milkie&quot;&gt;milkie&lt;/a&gt; good idea, and I think it&apos;s crucial, otherwise we will be setting an unncessarily low maximum rate of deletions. Consider the case where we have one active TTL collection and TTL has been disabled for a while (e.g. during live migration) and we have considerable catchup to do. Taking the example parameters mentioned above, we would be limited to 200 documents per second during the catchup period, and risk not ever catching up.&lt;/p&gt;</comment>
                            <comment id="3728174" author="milkie" created="Tue, 20 Apr 2021 16:55:12 +0000"  >&lt;p&gt;In addition to the limits, we might consider &quot;starting over from the top of the list&quot; after processing the last collection, if the TTL thread ever hits the limit for any collection or database, rather than stopping and waiting for the TTL period to expire.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1972351">SERVER-63040</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2039327">SERVER-66210</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="2056594">DOCS-15365</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2182763">SERVER-71290</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2058780">SERVER-66898</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1683965">SERVER-56195</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2052589">SERVER-66655</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2049336">SERVER-66537</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000xDjZHQA0, 5002K00000uEN8nQAG, 5002K00000zfmb2QAA, 5002K00000zcwA2QAI, 5006R00001md2WZQAY, 5006R00001xKpuuQAC]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 20 Apr 2021 16:55:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 36 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-63040'>SERVER-63040</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-66210'>SERVER-66210</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_17052" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Downstream Changes Summary</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Changes to serverStatus:&lt;br/&gt;
The TTL Monitor has a new ServerStatus field ttl.subPasses &lt;br/&gt;
&lt;br/&gt;
A given TTL pass may consist of 0 to many sub-passes. 0 only in the case the replica set is not in a readable state when the pass begins. &lt;br/&gt;
&lt;br/&gt;
The ServerStatus ttl.passes field remains the same in that a single pass deletes all expired documents (unless externally interrupted).&lt;br/&gt;
&lt;br/&gt;
Note: Behavior of the TTLMonitor only changes when server-parameter &amp;#39;ttlMonitorBatchDeletes&amp;#39; is set to true. Otherwise, the TTL Monitor uses legacy behavior and each TTL pass, provided the replica set is in a readable state, consists of a single sub-pass.&lt;br/&gt;
&lt;br/&gt;
New Behavior: &lt;br/&gt;
If the TTL monitor batches deletes (ttlMonitorBatchDeletes server param), then it provides fair TTL deletion as below.&lt;br/&gt;
- The TTL pass consists of zero, one or more subpasses.&lt;br/&gt;
- Each subpass deletes all expired documents on each TTL index in a round-robin fashion.&lt;br/&gt;
- The delete on each TTL index removes up to ttlIndexDeleteTargetDocs or runs up to ttlIndexDeleteTargetTimeMS, whichever happens first. The same TTL index can be queued up to be revisited in the same subpass if there are outstanding deletions.&lt;br/&gt;
- A TTL index is not visited any longer in a subpass once all documents are deleted.&lt;br/&gt;
- The duration of a subpass is limited to ttlMonitorSubPassTargetSecs. If there are outstanding deletions by the end of the subpass, a new subpass starts within the same pass.&lt;br/&gt;
&lt;br/&gt;
Motivation:&lt;br/&gt;
Legacy behavior involves single iteration over each TTL index, with an unbounded of documents removed and time spent executing deletes on a TTL index. Thus, the TTL Monitor could spend unbounded time deleting expired documents on one TTL index while starving the others from deletes.&lt;br/&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16942"><![CDATA[Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-2227</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 36 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>haley.connelly@mongodb.com</customfieldvalue>
            <customfieldvalue>louis.williams@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz5ewv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr23of:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="5117">Execution Team 2021-08-23</customfieldvalue>
    <customfieldvalue id="5979">Execution Team 2022-05-02</customfieldvalue>
    <customfieldvalue id="5980">Execution Team 2022-05-16</customfieldvalue>
    <customfieldvalue id="5981">Execution Team 2022-05-30</customfieldvalue>
    <customfieldvalue id="6029">Execution Team 2022-06-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_17051" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Teams Impacted</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16943"><![CDATA[Cloud]]></customfieldvalue>
    <customfieldvalue key="16944"><![CDATA[Docs]]></customfieldvalue>
    <customfieldvalue key="20960"><![CDATA[DBX: DevTools (Compass, Shell, VS Code Ext)]]></customfieldvalue>
    <customfieldvalue key="16946"><![CDATA[Triage and Release]]></customfieldvalue>
    <customfieldvalue key="20958"><![CDATA[Atlas Data Federation and Data Lake]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hz515z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>