<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:21:12 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-49916] use storageSize instead of size when creating capped collections</title>
                <link>https://jira.mongodb.org/browse/SERVER-49916</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When creating capped collections you have to define its maximal size in bytes which is size of &lt;b&gt;uncompressed&lt;/b&gt; data. &lt;br/&gt;
However, since wiredTiger stores data &lt;b&gt;compressed&lt;/b&gt; on disk, it is very difficult to guess the right &quot;size&quot; not to exceed a certain amount of data on disk because the compression factor (and thus the &quot;storageSize&quot;) may vary.&lt;/p&gt;

&lt;p&gt;Therefore, it would be much better to be able to define the maximal &quot;storageSize&quot; when creating a capped collection. For downwards compatibility, the &quot;size&quot; parameter can co-exists.&lt;/p&gt;

&lt;p&gt;This feature is even more important as long as capped collections cannot be resized.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1421334">SERVER-49916</key>
            <summary>use storageSize instead of size when creating capped collections</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="4" iconUrl="https://jira.mongodb.org/images/icons/priorities/minor.svg">Minor - P4</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="12300">Won&apos;t Do</resolution>
                                        <assignee username="michael.gargiulo@mongodb.com">Michael Gargiulo</assignee>
                                    <reporter username="kay.agahd@idealo.de">Kay Agahd</reporter>
                        <labels>
                    </labels>
                <created>Sun, 26 Jul 2020 20:47:57 +0000</created>
                <updated>Thu, 6 Apr 2023 15:39:08 +0000</updated>
                            <resolved>Fri, 18 Sep 2020 13:40:50 +0000</resolved>
                                    <version>3.6.17</version>
                    <version>4.0.19</version>
                    <version>4.2.8</version>
                                                    <component>Storage</component>
                                        <votes>1</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="4227368" author="alice@gothcandy.com" created="Thu, 2 Dec 2021 16:58:22 +0000"  >&lt;p&gt;I would agree with Kay, here. Compression adds variability to the storage requirements that is unfortunate. I did not actually anticipate compression being a factor with capped collections. In one of my logging collection cases, I requested 1GB, storage is consuming 182MB. This collection could be storing ~5.2&#215; as much historical information. (Currently dating back to&#160;2021-08-11, it could date back to ~2020-04-21.)&lt;/p&gt;

&lt;p&gt;My requirements for capped collections by size relates explicitly to on-disk storage limitations / allocation. Pathological compression cases (I&apos;ve identified a few, zlib and Huffman compression schemes are funny this way) &lt;em&gt;could&lt;/em&gt; result in on-disk storage consuming significantly more space than the input data as a result of this, a potential DOS vector. (Example bad, but at least not a growth case: &lt;a href=&quot;https://stackoverflow.com/questions/43666802/efficiently-encoding-rle-an-array-of-generally-consecutive-line-numbers&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;compressing a stream of zeros or monotonic integers&lt;/a&gt;. In Zlib, this results in an ironically highly compressible and growing stream of `A` in the output.)&lt;/p&gt;

&lt;p&gt;Finally, TTL is not a replacement for a ring buffer.&#160;&lt;b&gt;At all.&lt;/b&gt; TTL sweeps happen per-minute, might not complete, and thus &lt;a href=&quot;https://github.com/marrow/mongo/blob/develop/marrow/mongo/core/trait/expires.py?ts=4#L41&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;require explicit validation application-side&lt;/a&gt; even if used. A TTL sweep might be fine for &quot;eventual consistency of storage reclamation&quot;, it&apos;s not so good for hard guarantees.&lt;/p&gt;</comment>
                            <comment id="3401912" author="kay.agahd@idealo.de" created="Fri, 18 Sep 2020 22:39:03 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.gargiulo&quot; class=&quot;user-hover&quot; rel=&quot;michael.gargiulo&quot;&gt;michael.gargiulo&lt;/a&gt;, &lt;br/&gt;
thank you for the information, even though I believe that your team made the wrong decision in not supporting capped collections in the future. &lt;/p&gt;

&lt;p&gt;If you want to use 80% of your available disk space to store historical data, how would you do that with TTL indexing? It&apos;s just not possible! So please renounce your decision and support capped collections by letting the user define its maximal &quot;storageSize&quot; as I&apos;ve written in the description.&lt;br/&gt;
Thanks!&lt;/p&gt;</comment>
                            <comment id="3400771" author="michael.gargiulo" created="Fri, 18 Sep 2020 13:40:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kay.agahd%40idealo.de&quot; class=&quot;user-hover&quot; rel=&quot;kay.agahd@idealo.de&quot;&gt;kay.agahd@idealo.de&lt;/a&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thank you for sharing this blog post, it was helpful to understand your use cases and how you are using capped collections. &lt;/p&gt;

&lt;p&gt;I wanted to let you know that we have made a preliminary decision to stop supporting user created capped collections in an undetermined future release of MongoDB in favor of improving TTL indexes so they can meet the needs of users who currently employ capped collections. While we have no current timeline to discontinue this support, we will not be doing new feature requests related to capped collections so I will be closing this ticket and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-49917&quot; title=&quot;allow to resize capped collections &quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-49917&quot;&gt;&lt;del&gt;SERVER-49917&lt;/del&gt;&lt;/a&gt;.&#160;&lt;/p&gt;

&lt;p&gt;Since you mentioned TTL indexes are not ideal for your scenario, as you must choose an index, I would love to hear more from you so we keep the use cases in mind when moving forward with designing how we better improve upon the concept of user created capped collections and improve TTL indexing.&#160;&lt;br/&gt;
&#160;&lt;/p&gt;</comment>
                            <comment id="3325544" author="kay.agahd@idealo.de" created="Thu, 6 Aug 2020 21:00:19 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;I just wrote in our techblog how we worked around the issue that capped collections cannot be resized without downtime so far:&lt;br/&gt;
&lt;a href=&quot;https://medium.com/idealo-tech-blog/mission-possible-resize-mongodb-capped-collections-without-downtime-ec8aada2223f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://medium.com/idealo-tech-blog/mission-possible-resize-mongodb-capped-collections-without-downtime-ec8aada2223f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="3308725" author="dmitry.agranat" created="Tue, 28 Jul 2020 05:00:27 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kay.agahd%40idealo.de&quot; class=&quot;user-hover&quot; rel=&quot;kay.agahd@idealo.de&quot;&gt;kay.agahd@idealo.de&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;Thank you for clarifying your request is not related to Oplog but for other on-demand created capped collections. I am passing this request to one of our teams for review.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                            <comment id="3307683" author="kay.agahd@idealo.de" created="Mon, 27 Jul 2020 16:27:09 +0000"  >&lt;p&gt;Hi Dima,&lt;/p&gt;

&lt;p&gt;you got me wrong. I was not talking about the oplog. I&apos;m happy how the oplog works since mongodb v3.6 (because we can resize it on the fly) and also since v4.0.&lt;br/&gt;
My feature request is for all other capped collections, not the oplog. I hoped that the issues that I&apos;ve linked made this even more clear.&lt;/p&gt;</comment>
                            <comment id="3307017" author="dmitry.agranat" created="Mon, 27 Jul 2020 12:19:16 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kay.agahd%40idealo.de&quot; class=&quot;user-hover&quot; rel=&quot;kay.agahd@idealo.de&quot;&gt;kay.agahd@idealo.de&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Let&apos;s assume we are talking about Oplog (which is capped collection) and not other types of capped collections. Starting in MongoDB 4.0, the Oplog can grow past its configured size limit to avoid deleting the majority commit point so I am not sure how limiting Oplog to one metric or another would help you with setting the &quot;right&quot; size which you will never have to change. Apart, Oplog resize method can be used if your write workload changes and you need to increase it based on the maintenance window requirements.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dima&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1421337">SERVER-49918</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="1421335">SERVER-49917</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 27 Jul 2020 12:19:16 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 9 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>false</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 9 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alice@gothcandy.com</customfieldvalue>
            <customfieldvalue>dmitry.agranat@mongodb.com</customfieldvalue>
            <customfieldvalue>kay.agahd@idealo.de</customfieldvalue>
            <customfieldvalue>michael.gargiulo@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxwuan:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hxlplb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="4126">Execution Team 2020-08-24</customfieldvalue>
    <customfieldvalue id="4127">Execution Team 2020-09-07</customfieldvalue>
    <customfieldvalue id="4213">Execution Team 2020-09-21</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[dmitry.agranat@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hxwgjz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>