<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:11:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26131] MongoDB, XFS, and SSDs</title>
                <link>https://jira.mongodb.org/browse/SERVER-26131</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have run into an issue with XFS&#8217;s &lt;tt&gt;FITRIM&lt;/tt&gt; &lt;tt&gt;ioctl&lt;/tt&gt; implementation (see: &lt;a href=&quot;https://github.com/torvalds/linux/blob/master/fs/xfs/xfs_discard.c#L155&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/torvalds/linux/blob/master/fs/xfs/xfs_discard.c#L155&lt;/a&gt;) (used by the &lt;tt&gt;fstrim&lt;/tt&gt; command (see: &lt;a href=&quot;https://github.com/karelzak/util-linux/blob/master/sys-utils/fstrim.c#L87&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/karelzak/util-linux/blob/master/sys-utils/fstrim.c#L87&lt;/a&gt;)) when running against local SSDs that is severely impacting IO in general and MongoDB specifically.&lt;/p&gt;

&lt;p&gt;Essentially, XFS is iterating over every allocation group and issuing &lt;tt&gt;TRIM&lt;/tt&gt; s for all free extents &lt;b&gt;every&lt;/b&gt; time this &lt;tt&gt;ioctl&lt;/tt&gt; is called. This, coupled with the facts that Linux&#8217;s interface to the &lt;tt&gt;TRIM&lt;/tt&gt; command is both synchronous and does not support a vectorized list of ranges (see: &lt;a href=&quot;https://github.com/torvalds/linux/blob/3fc9d690936fb2e20e180710965ba2cc3a0881f8/block/blk-lib.c#L112&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/torvalds/linux/blob/3fc9d690936fb2e20e180710965ba2cc3a0881f8/block/blk-lib.c#L112&lt;/a&gt;), is leading to a large number of extraneous &lt;tt&gt;TRIM&lt;/tt&gt; commands (each of which have been observed to be slow, see: &lt;a href=&quot;http://oss.sgi.com/archives/xfs/2011-12/msg00311.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://oss.sgi.com/archives/xfs/2011-12/msg00311.html&lt;/a&gt;) being issued to the disk for ranges that both the filesystem and the disk know to be free. In practice, we have seen IO disruptions of up to 2 minutes. I realize that the duration of these disruptions may be controller dependent. Unfortunately, when running on a platform like AWS, one does not have the luxury of choosing specific hardware.&lt;/p&gt;

&lt;p&gt;EXT4, on the other hand, tracks blocks that have been deleted since the previous &lt;tt&gt;FITRIM&lt;/tt&gt; ioctl and targets subsequent &lt;tt&gt;TRIM&lt;/tt&gt; s to the appropriate block ranges (see: &lt;a href=&quot;http://blog.taz.net.au/2012/01/07/fstrim-and-xfs/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://blog.taz.net.au/2012/01/07/fstrim-and-xfs/&lt;/a&gt;). In real-world tests this significantly reduces the impact of &lt;tt&gt;fstrim&lt;/tt&gt; to the point that it is un-noticeable to the database / application. We are currently switching back to EXT4 as a result.&lt;/p&gt;

&lt;p&gt;Alternatively, we could mount the filesystem with the &lt;tt&gt;discard&lt;/tt&gt; option (as AWS suggests here: &lt;a href=&quot;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html&lt;/a&gt;), however, our confidence in this performing better is not high given XFS developer comments on the subject (see: &lt;a href=&quot;http://oss.sgi.com/archives/xfs/2014-08/msg00465.html):&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://oss.sgi.com/archives/xfs/2014-08/msg00465.html):&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;It was introduced into XFS as a checkbox feature. We resisted as&lt;br/&gt;
long as we could, but too many people were shouting at us that we&lt;br/&gt;
needed realtime discard because ext4 and btrfs had it. Of course,&lt;br/&gt;
all those people shouting for it realised that we were right in that&lt;br/&gt;
it sucked the moment they tried to use it and found that performance&lt;br/&gt;
was woeful. Not to mention that SSD trim implementations were so bad&lt;br/&gt;
that they caused random data corruption by trimming the wrong&lt;br/&gt;
regions, drives would simply hang randomly and in a couple of cases&lt;br/&gt;
too many trims too fast would brick them...&lt;/p&gt;

&lt;p&gt;So, yeah, it was implement because lots of people demanded it, not&lt;br/&gt;
because it was a good idea.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I am aware that MongoDB &lt;b&gt;strongly&lt;/b&gt; recommends using XFS (see: &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/#kernel-and-file-systems&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.mongodb.com/manual/administration/production-notes/#kernel-and-file-systems&lt;/a&gt;) and that this is because EXT4 journaling could impact Wired Tiger checkpointing under heavy write load (&lt;a href=&quot;https://groups.google.com/forum/#!msg/mongodb-user/diGdooN_2Sw/4H7t5JTDcpAJ&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/#!msg/mongodb-user/diGdooN_2Sw/4H7t5JTDcpAJ&lt;/a&gt;). Can you elaborate on this? Is this the only concern that drove the &lt;b&gt;strong&lt;/b&gt; recommendation to go with XFS and, in MongoDB&#8217;s opinion, is this still valid given the performance issues with &lt;tt&gt;TRIM&lt;/tt&gt; on Linux when running XFS on SSDs? We are currently running the MMAPv1 storage engine on MongoDB 2.6 and, as mentioned above, we have reverted to EXT4 without apparent consequence. Any more info that you could provide would really help us in weighing the pros and cons while we work toward Wired Tiger.&lt;/p&gt;

&lt;p&gt;Also, any more general recommendations for mitigating the disruption incurred by running &lt;tt&gt;fstrim&lt;/tt&gt; would be more than welcome.&lt;/p&gt;</description>
                <environment></environment>
        <key id="316579">SERVER-26131</key>
            <summary>MongoDB, XFS, and SSDs</summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="gregbanks">Gregory Banks</reporter>
                        <labels>
                    </labels>
                <created>Thu, 15 Sep 2016 21:28:34 +0000</created>
                <updated>Tue, 20 Sep 2016 16:35:33 +0000</updated>
                            <resolved>Fri, 16 Sep 2016 18:06:31 +0000</resolved>
                                    <version>2.6.12</version>
                    <version>3.0.12</version>
                    <version>3.2.9</version>
                                                    <component>Performance</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1388813" author="gregbanks" created="Tue, 20 Sep 2016 16:35:33 +0000"  >&lt;p&gt;&lt;a href=&quot;https://groups.google.com/forum/#!topic/mongodb-user/Mj0x6m-02Ms&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/forum/#!topic/mongodb-user/Mj0x6m-02Ms&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1386721" author="gregbanks" created="Fri, 16 Sep 2016 20:59:46 +0000"  >&lt;p&gt;Thanks Thomas. I&apos;ll move discussion over to the group.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;br/&gt;
Greg&lt;/p&gt;</comment>
                            <comment id="1386539" author="thomas.schubert" created="Fri, 16 Sep 2016 18:06:31 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=gregbanks&quot; class=&quot;user-hover&quot; rel=&quot;gregbanks&quot;&gt;gregbanks&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed question. We recommend XFS since we have observed long pauses related to EXT4. However, if you have tested your workload with WiredTiger on EXT4 and see better results, then I don&apos;t see a reason why you can&apos;t move forward with it.&lt;/p&gt;

&lt;p&gt;Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the &lt;a href=&quot;http://groups.google.com/group/mongodb-user&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;mongodb-users group&lt;/a&gt; or &lt;a href=&quot;http://stackoverflow.com/questions/tagged/mongodb&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Stack Overflow with the &lt;tt&gt;mongodb&lt;/tt&gt; tag&lt;/a&gt;. A question like this involving more discussion would be best posted on the mongodb-users group.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 16 Sep 2016 18:06:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 21 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>gregbanks</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 21 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>gregbanks</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjvzj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hspt8v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsejkf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>