<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:42:03 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-16724] Startup time under WiredTiger is extremely slow and gets worse as database grows</title>
                <link>https://jira.mongodb.org/browse/SERVER-16724</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I did a test of my company&apos;s analytics app, using the WiredTiger storage engine, on Windows Server 2012, on a HyperV VM. During the test, I noticed that startup time for MongoDB was getting excessive - 2 hours in one case, 2.5 hours in another. It had been very quick the first few times I started it. I didn&apos;t take a lot of time to log the details of what was wrong, but the difference between the various times was only in the amount of data in the database - the more data, the slower the startup time. (Roughly 6MM records -&amp;gt; 2 hours, roughly 7.5MM -&amp;gt; 2.5 hours.)&lt;/p&gt;

&lt;p&gt;I don&apos;t have stats from those instances, but one of the databases in the instance is a single collection with one record per incoming message, stored as binary data, with a few fields of metadata and a few indexes. I assume it&apos;s that database that&apos;s causing the problem. (During the entire startup period, the system shows lots of read activity for a single huge file from the mongodb data directory.) The other databases have much lower overall volume/size.&lt;/p&gt;

&lt;p&gt;Sometime in the next day or so I&apos;ll need to restart it again, with about 20MM records in it. I&apos;ll collect more detailed info about it, then.&lt;/p&gt;

&lt;p&gt;I&apos;ve attached log snippets from the two slow startups.&lt;/p&gt;</description>
                <environment></environment>
        <key id="176992">SERVER-16724</key>
            <summary>Startup time under WiredTiger is extremely slow and gets worse as database grows</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.benvenuto@mongodb.com">Mark Benvenuto</assignee>
                                    <reporter username="narthur_pre">Nathan Arthur</reporter>
                        <labels>
                    </labels>
                <created>Mon, 5 Jan 2015 18:08:09 +0000</created>
                <updated>Thu, 15 Jan 2015 16:46:34 +0000</updated>
                            <resolved>Tue, 13 Jan 2015 16:35:45 +0000</resolved>
                                    <version>2.8.0-rc3</version>
                                    <fixVersion>2.8.0-rc5</fixVersion>
                                    <component>Performance</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="803454" author="narthur_pre" created="Tue, 13 Jan 2015 16:46:46 +0000"  >&lt;p&gt;Fantastic - thank you!&lt;/p&gt;</comment>
                            <comment id="803440" author="mark.benvenuto" created="Tue, 13 Jan 2015 16:35:29 +0000"  >&lt;p&gt;I believe I was able to create a repro of your bug by doing the following:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Start server with WT&lt;/li&gt;
	&lt;li&gt;Insert documents into server for a while (I had 10 GB)&lt;/li&gt;
	&lt;li&gt;Terminate MongoD process. I used &lt;tt&gt;kill.exe&lt;/tt&gt; to terminate the MongoD process abruptly, Control-C would have done a clean shutdown&lt;/li&gt;
	&lt;li&gt;Start MongoD again, and MongoD startup takes a long time (I stopped it after waiting for 10 minutes).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I fix has been pushed to master, and I believe this will resolve the issue you saw.&lt;/p&gt;
</comment>
                            <comment id="803409" author="xgen-internal-githook" created="Tue, 13 Jan 2015 16:00:20 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;username&apos;: u&apos;markbenvenuto&apos;, u&apos;name&apos;: u&apos;Mark Benvenuto&apos;, u&apos;email&apos;: u&apos;mark.benvenuto@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16724&quot; title=&quot;Startup time under WiredTiger is extremely slow and gets worse as database grows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16724&quot;&gt;&lt;del&gt;SERVER-16724&lt;/del&gt;&lt;/a&gt;: fix syncSizeStorer&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/60dc6bf36715570fbf0287d83c67e60b5f408baa&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/60dc6bf36715570fbf0287d83c67e60b5f408baa&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="803057" author="narthur_pre" created="Tue, 13 Jan 2015 01:08:36 +0000"  >&lt;p&gt;This seems like a plausible explanation, but there shouldn&apos;t have been any new collections created during these runs. The server had been started and restarted multiple times, and was even restarted after it already had lots of data, multiple times. The only way I can think that this would make sense is if every single shutdown was failing, and the problem propagated across starts - but that seems unlikely.&lt;/p&gt;</comment>
                            <comment id="802969" author="mark.benvenuto" created="Mon, 12 Jan 2015 23:16:31 +0000"  >&lt;p&gt;In this scenario, due to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16106&quot; title=&quot;Event Viewer (system) error when pressing power button on machine&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16106&quot;&gt;&lt;del&gt;SERVER-16106&lt;/del&gt;&lt;/a&gt;, the server was abruptly terminated.&lt;/p&gt;

&lt;p&gt;If the server is abruptly terminated during a session when a collection has just been created, then the server does not go through a clean shutdown. On start up, WiredTigerRecordStore::WiredTigerRecordStore counts every record in the collection, and gets the size of each record.&lt;/p&gt;</comment>
                            <comment id="798531" author="narthur_pre" created="Tue, 6 Jan 2015 22:35:33 +0000"  >&lt;p&gt;I attached a new file with a bunch of shutdown logs. I didn&apos;t see any of those messages you listed, in any of them. (Even ones where the subsequent startup was fast.)  Are you sure about that logging, on Windows, on RC3?&lt;/p&gt;

&lt;p&gt;Note that in the slow startup scenarios, nothing was logged about it doing a recovery - it just said it was starting up, then hung for hours, then started accepting connections.&lt;/p&gt;

&lt;p&gt;I also just tried hard-killing the mongod.exe process, and it started up immediately again.&lt;/p&gt;</comment>
                            <comment id="798508" author="mark.benvenuto" created="Tue, 6 Jan 2015 22:11:42 +0000"  >&lt;p&gt;In your attachment &quot;additional_startup_time_logs.txt&quot;,  do you trim any of the output from &quot;&lt;tt&gt;PRIOR SHUTDOWN&lt;/tt&gt;&quot;. I expect something like the following:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-01-06T17:00:20.050-0500 I NETWORK  [serviceShutdown] shutdown: going to close sockets...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-01-06T17:00:20.050-0500 I STORAGE  [serviceShutdown] WiredTigerKVEngine shutting down&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-01-06T17:00:20.101-0500 I STORAGE  [serviceShutdown] shutdown: removing fs lock...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-01-06T17:00:20.101-0500 I COMMAND  [serviceShutdown] dbexit:  rc: 49&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;If you did not trim the shutdown log output, this means that Windows completed the shutdown before MongoDB had time to finish cleanly shutting down. This means Windows killed mongod.exe, and WiredTiger then had to go through recovery on restart.&lt;/p&gt;

&lt;p&gt;I will work in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-16106&quot; title=&quot;Event Viewer (system) error when pressing power button on machine&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-16106&quot;&gt;&lt;del&gt;SERVER-16106&lt;/del&gt;&lt;/a&gt; to improve our Shutdown handler, and ensure &lt;a href=&quot;http://technet.microsoft.com/en-us/library/cc976045.aspx&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;WaitToKillServiceTimeout&lt;/a&gt; is set correctly.&lt;/p&gt;</comment>
                            <comment id="798013" author="narthur_pre" created="Tue, 6 Jan 2015 14:14:58 +0000"  >&lt;p&gt;...and of course now it starts up essentially instantly. Even after multiple restarts. I have no idea why.&lt;/p&gt;

&lt;p&gt;There have been NO configuration changes to this host; the only difference is a lot more data imported (via application code), and a lot more time passed.&lt;/p&gt;

&lt;p&gt;I found a third log (from before) where startup time was around 4 hours. I don&apos;t remember the database size at the time, but it would have been correspondingly larger then the earlier examples.  I&apos;ll attach that, as well as the log snippit from the prior shutdown (which was perfectly normal). &lt;/p&gt;</comment>
                            <comment id="798005" author="narthur_pre" created="Tue, 6 Jan 2015 14:05:15 +0000"  >&lt;p&gt;I&apos;m about to restart the server again. I collected a bunch of stats, which I&apos;ll upload as attachments (one with commands, one with the output). I&apos;ll report back later with the total restart time.&lt;/p&gt;</comment>
                            <comment id="797989" author="dan@10gen.com" created="Tue, 6 Jan 2015 13:24:23 +0000"  >&lt;p&gt;Hi Nathan, thanks for the report.  Can you run &lt;tt&gt;db.stats()&lt;/tt&gt; and &lt;tt&gt;db.collection.stats()&lt;/tt&gt; (substituting your collection name here) and add to this ticket?&lt;/p&gt;

&lt;p&gt;Also, how many total databases and collections do you have on this instance?&lt;/p&gt;</comment>
                            <comment id="797277" author="narthur_pre" created="Mon, 5 Jan 2015 18:10:07 +0000"  >&lt;p&gt;I just noticed from the logs - apparently this was actually RC3, not RC0 as I tagged it.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="60986" name="additional_startup_time_logs.txt" size="3231" author="narthur_pre" created="Tue, 6 Jan 2015 14:16:27 +0000"/>
                            <attachment id="60980" name="db_details_commands.txt" size="380" author="narthur_pre" created="Tue, 6 Jan 2015 14:04:52 +0000"/>
                            <attachment id="60981" name="db_details_output.txt" size="30979" author="narthur_pre" created="Tue, 6 Jan 2015 14:04:52 +0000"/>
                            <attachment id="60876" name="mongodb_startup_wiredtiger_slow.txt" size="3482" author="narthur_pre" created="Mon, 5 Jan 2015 18:08:09 +0000"/>
                            <attachment id="61046" name="more_shutdown_logs.txt" size="5119" author="narthur_pre" created="Tue, 6 Jan 2015 22:33:03 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 6 Jan 2015 13:24:23 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 5 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ernie.hershey@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 5 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>narthur_pre</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkx07:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs56k7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>155240</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs2brr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>