<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:14:06 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7287] --tinyfiles (32M) option for effective GridFS datafile caching on large file server</title>
                <link>https://jira.mongodb.org/browse/SERVER-7287</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently --smallfiles option lets GridFS save file chunks into 512MB datafiles, this is not small enough for effective read caching on ZFS, and is causing problems on our ZFS+GridFS file server:&lt;/p&gt;

&lt;p&gt;1. Our GridFS server runs on ZFS, total data size is ~30TB, Ram is 64G.&lt;/p&gt;

&lt;p&gt;2. Using the --smallfiles option, GridFS saves ~30TB, ~12 million files as chunks in ~60,000 x 512MB datafiles.&lt;/p&gt;

&lt;p&gt;3. ZFS first uses RAM as file read cache, when RAM is filled up, ZFS then uses SSD as file read cache (L2ARC).&lt;/p&gt;

&lt;p&gt;4. Our application has ~30,000 hot files being fetched frequently at random hours.&lt;/p&gt;

&lt;p&gt;5. GridFS spreads the chunks of these 30,000 files across 1,000+ 512MB datafiles.&lt;/p&gt;

&lt;p&gt;6. Using a 256GB SSD as read cache for ZFS, ZFS only has enough space to cache ~460 GridFS datafiles of 512MB each.&lt;/p&gt;

&lt;p&gt;7. Excessive mechanical disk seeks are&#12288;caused by the application making requests to GridFS for chunks in datafiles not cached by ZFS.&lt;/p&gt;

&lt;p&gt;8. Adding more Ram does almost nothing to help the situation, using a bigger (512GB) SSD is not cost effective and will still cause disk seeks when a file chunk is saved in the ~921th datafile.&lt;/p&gt;

&lt;p&gt;9. If --tinyfiles enables GridFS to save chunks into 32MB datafiles, then a 256GB SSD can now cache ~7400 datafiles which contains all the hot chunks, and reduce mechanical disk seek to almost zero.&lt;/p&gt;

&lt;p&gt;Bottom line: Tiny (32MB) datafile size may not make a difference for normal MongoDB data, but is critical for large GridFS servers on ZFS.&lt;/p&gt;
</description>
                <environment>ZFS</environment>
        <key id="52529">SERVER-7287</key>
            <summary>--tinyfiles (32M) option for effective GridFS datafile caching on large file server</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mathias@mongodb.com">Mathias Stearn</assignee>
                                    <reporter username="alex_yam">Alex Yam</reporter>
                        <labels>
                            <label>Cache</label>
                            <label>GridFS</label>
                            <label>ZFS</label>
                            <label>chunks</label>
                            <label>datafiles</label>
                    </labels>
                <created>Sun, 7 Oct 2012 04:56:36 +0000</created>
                <updated>Fri, 15 Feb 2013 15:06:13 +0000</updated>
                            <resolved>Wed, 10 Oct 2012 19:12:27 +0000</resolved>
                                    <version>2.2.0</version>
                                                    <component>GridFS</component>
                    <component>Performance</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="173424" author="alex_yam" created="Wed, 10 Oct 2012 17:35:27 +0000"  >&lt;p&gt;After a few days of head scratching we finally found the problem, the HDD seeks were caused by Nginx deleting old cache files and writing new ones (LRU replacement policy when the cache is full), we missed this setting when we merged the Nginx config files from different servers, as a result large files were sharing the same small cache zone with the avatars/images.&lt;/p&gt;

&lt;p&gt;Took a while to find the problem because the blinkings were inconsistent, there were 4 different caches in play, some get flushed after reboot but some don&apos;t:&lt;br/&gt;
1. ram cache for MongoDB&lt;br/&gt;
2. ram cache for ZFS&lt;br/&gt;
3. SSD cache for ZFS&lt;br/&gt;
4. HDD cache for Nginx&lt;/p&gt;

&lt;p&gt;Problem was solved by placing images/avatars and large files in different Nginx cache zones, after the change, no more disk seeks for images/avatars after hammering the large files server.&lt;/p&gt;

&lt;p&gt;Hope this ticket can help others who come across the same situation.&lt;/p&gt;</comment>
                            <comment id="172352" author="alex_yam" created="Sun, 7 Oct 2012 11:06:07 +0000"  >&lt;p&gt;If ZFS cache is block based then we must have misconfigured something.&lt;/p&gt;

&lt;p&gt;We recently merged multiple GridFS DBs into a new ZFS server to improve reliability, when we stress tested the GridFS server, we discovered that the HDD lights on the hot swap bays were blinking nonstop.&lt;/p&gt;

&lt;p&gt;Stress tests were done using http_load and 2 URL lists:&lt;br/&gt;
1st list consists of small files, with URLs equivalent to &lt;a href=&quot;http://image_server/gridfs_fetch.php?filename=avatar_&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://image_server/gridfs_fetch.php?filename=avatar_&lt;/a&gt;{$userid}.jpg&lt;br/&gt;
2nd list consists of larger/huge files, with URLs equivalent to &lt;a href=&quot;http://download_server/gridfs_fetch.php?filename=&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://download_server/gridfs_fetch.php?filename=&lt;/a&gt;{$file_name}_{$file_id}.{$file_ext}&lt;/p&gt;

&lt;p&gt;OS is running FreeBSD9, HTTP is Nginx+PHP-FPM, ZFS zpool contains a raidz3 vdev using 11 x 3TB disks, 8 of which are connected to a HBA and 3 connected to on board sata ports.&lt;/p&gt;

&lt;p&gt;Our test procedures:&lt;br/&gt;
1. Reboot to flush all caches&lt;br/&gt;
2. Run stress test with 1st URL list multiple times (avatar and images URLs) &amp;#8212; HDD lights on the hot swap bays blink in the first run, but not in the second/third run.&lt;br/&gt;
3. Run stress test with 2nd URL list (download URLs) &amp;#8212; HDD lights blink for as long as it runs.&lt;br/&gt;
4. Run stress test with 1st URL list again &amp;#8212; HDD lights blink again, but no blinking at second/third run.&lt;/p&gt;

&lt;p&gt;Running &quot;zpool iostat -v&quot; from the shell shows the read cache SSD only has 16M free space left so the cache is working.&lt;/p&gt;

&lt;p&gt;We are new to ZFS, so when the HDD lights blink at step 4, we assumed this was caused by ZFS caching entire GridFS datafiles at step 3 and cleared the cache at step 2.&lt;/p&gt;

&lt;p&gt;The blinking HDD lights tell us if the system goes live, the random avatar/images fetches will wear out our disks more than necessary, this is where we are stuck at the moment.&lt;/p&gt;

&lt;p&gt;Are there tools we can use to pin point exactly what files are being accessed?&lt;/p&gt;</comment>
                            <comment id="172330" author="eliot" created="Sun, 7 Oct 2012 05:06:22 +0000"  >&lt;p&gt;Filesystems (zfs) included don&apos;t have to cache the entire file, they can cache only the pages they need.&lt;br/&gt;
Do you see evidence of something else here?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 7 Oct 2012 05:06:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 19 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 19 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alex_yam</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>mathias@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnlyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrjv5b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>29580</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht06db:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>