<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:58 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-27036] WT takes longer to start than MMAPv1 on EBS volumes that have not been intialized</title>
                <link>https://jira.mongodb.org/browse/SERVER-27036</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hey,&lt;/p&gt;

&lt;p&gt;We recently upgraded from MMAP to WT. For the same data amount (~500GB), on an AWS EBS volume which is &quot;cold&quot; (restored from a snapshot) - size 2TB -&lt;br/&gt;
MMAP took ~5 minutes to start&lt;br/&gt;
WT takes 50 &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/warning.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; minutes, log mentioning this:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Taking 1005 samples and assuming that each section of oplog contains approximately 5672862 records totaling to 1073741955 bytes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;It apparently scans the oplog (we&apos;ve set its size to 100GB) - and we see the iowait on 100%. But still, 50 minutes is a lot...&lt;/p&gt;

&lt;p&gt;Anything we can do to speed things up?&lt;/p&gt;</description>
                <environment></environment>
        <key id="331876">SERVER-27036</key>
            <summary>WT takes longer to start than MMAPv1 on EBS volumes that have not been intialized</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="yonido">Yoni Douek</reporter>
                        <labels>
                    </labels>
                <created>Tue, 15 Nov 2016 08:29:34 +0000</created>
                <updated>Mon, 17 Jun 2019 09:50:06 +0000</updated>
                            <resolved>Mon, 21 Nov 2016 22:51:28 +0000</resolved>
                                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>18</watches>
                                                                                                                <comments>
                            <comment id="1439572" author="michael.cahill" created="Mon, 21 Nov 2016 22:50:39 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yonido&quot; class=&quot;user-hover&quot; rel=&quot;yonido&quot;&gt;yonido&lt;/a&gt;, as explained above, MongoDB 3.2 samples the oplog to optimize truncation with WiredTiger.  This process isn&apos;t inherently slow, but happens during startup so you may notice it more than slow reads via MMAPv1.&lt;/p&gt;

&lt;p&gt;I&apos;m going to close this now, the issue with slow reads from EBS is understood and documented by AWS, there isn&apos;t much we can do about it in MongoDB.&lt;/p&gt;</comment>
                            <comment id="1438733" author="yonido" created="Mon, 21 Nov 2016 06:15:57 +0000"  >&lt;p&gt;Yep, this will be a workaround. But I though you guys might be interested to see why it&apos;s x10 slower than mmap.&lt;/p&gt;</comment>
                            <comment id="1438619" author="michael.cahill" created="Mon, 21 Nov 2016 03:16:08 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoni%40appsee.com&quot; class=&quot;user-hover&quot; rel=&quot;yoni@appsee.com&quot;&gt;yoni@appsee.com&lt;/a&gt;, if a warmup of the full EBS volume is too slow, one thing you could consider is following the AWS instructions to sequentially read through just the files in the database path for MongoDB.&lt;/p&gt;

&lt;p&gt;In fact, if you get the filename from the &lt;tt&gt;wiredtiger.uri&lt;/tt&gt; collection stats for the oplog, you could just fault in that file before starting MongoDB to avoid waiting on EBS.&lt;/p&gt;</comment>
                            <comment id="1436290" author="yonido" created="Thu, 17 Nov 2016 06:32:33 +0000"  >&lt;p&gt;Thanks.&lt;br/&gt;
Attached. In this example it took ~15 min but sometimes it takes much more.&lt;/p&gt;

&lt;p&gt;The EBS warm up process was taken into consideration when opening this ticket - we are aware that this affects the performance drastically. But when comparing cold disks in WT vs mmap - this is 10x slower. We are sure that pre-warming the EBS drive will help, but unfortunately this process make take several hours on large disks.&lt;/p&gt;
</comment>
                            <comment id="1436142" author="michael.cahill" created="Wed, 16 Nov 2016 23:31:44 +0000"  >&lt;p&gt;Also, one cause of very slow I/O when starting MongoDB on an EBS volume created from a snapshot is described in the &lt;a href=&quot;http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;AWS docs Initialising Amazon EBS Volumes&lt;/a&gt;,&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;For a new volume created from a snapshot, you should read all the blocks that have data before using the volume.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;If this applies to you, please try one of the steps recommended by AWS in the link above to see if that resolves the slow I/O.&lt;/p&gt;</comment>
                            <comment id="1436132" author="michael.cahill" created="Wed, 16 Nov 2016 23:12:10 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=yoni%40appsee.com&quot; class=&quot;user-hover&quot; rel=&quot;yoni@appsee.com&quot;&gt;yoni@appsee.com&lt;/a&gt;, sorry to hear that you are experiencing this problem.&lt;/p&gt;

&lt;p&gt;During this part of startup, MongoDB takes samples of the oplog, which involves random lookups in the WiredTiger btree that contains the oplog.  Each lookup usually results in reading a single leaf page, so here that would mean about 1005 page reads.  Even with slow spinning disks, that is not expected to take more than about 10 seconds.&lt;/p&gt;

&lt;p&gt;Can you please attach the &lt;tt&gt;mongod&lt;/tt&gt; log file (at least the section covering startup until this process is complete) and an archive of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory from your database path?  That is a log of statistics that should help us understand whether there is something we can change in MongoDB to make this process faster.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="804926">DOCS-12807</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="804917">SERVER-41790</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="144351" name="log" size="12995" author="yonido" created="Thu, 17 Nov 2016 06:32:25 +0000"/>
                            <attachment id="144350" name="tmp.zip" size="919876" author="yonido" created="Thu, 17 Nov 2016 06:32:26 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000VEHxFIAX]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 15 Nov 2016 10:14:13 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 12 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>marco.bonezzi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 12 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
            <customfieldvalue>yonido</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjqv3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsrvyf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsecnz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>