<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:49:02 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18887] WiredTiger instances taking 2 hours to startup and transition to Secondary state</title>
                <link>https://jira.mongodb.org/browse/SERVER-18887</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;WiredTiger instances are taking a long time to startup to the point where it becomes a Secondary or Primary. The log output up to where the long delay occurs before the transition to the Secondary/Primary state is shown below:&lt;/p&gt;

&lt;p&gt;2015-06-09T00:41:34.920+0000 I CONTROL  ***** SERVER RESTARTED *****&lt;br/&gt;
2015-06-09T00:41:35.206+0000 I CONTROL  Trying to start Windows service &apos;MongoDB&apos;&lt;br/&gt;
2015-06-09T00:41:35.224+0000 I STORAGE  Service running&lt;br/&gt;
2015-06-09T00:41:35.254+0000 W -        &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Detected unclean shutdown - m:/mongodb/data\mongod.lock is not empty.&lt;br/&gt;
2015-06-09T00:41:35.254+0000 W STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Recovering data from the last clean checkpoint.&lt;br/&gt;
2015-06-09T00:41:35.255+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),&lt;/p&gt;

&lt;p&gt;During this time MongoDB appears to be reading every file in our database, we guess its some kind of file integrity check. I&apos;m guessing this might be normal following an unclean shutdown in which case we need help with the shutdown issue . We&apos;ve noticed a few Jira Issues relating to abnormal shutdowns due to the Windows Service manager and timeouts but they appear to be getting marked as fixed but as of yet our system keeps getting this problem. I have tried shutting down the service from a command prompt as well as the service manager and normal OS restarts but every one results in a long startup time. I&apos;m not certain they all have produced the &apos;unclean shutdown&apos; log message though.&lt;/p&gt;

&lt;p&gt;We are running a 5 node replica set, 2 MMAP, 2 WT and 1 arbiter. Our database size is ~280GB (MMAP) / 180GB (WT). We were planning to transition to WiredTiger but this and a few other problems are showstoppers for us at the moment.&lt;/p&gt;</description>
                <environment>Windows 2012 R2 VM running in Azure, 7GB RAM</environment>
        <key id="209372">SERVER-18887</key>
            <summary>WiredTiger instances taking 2 hours to startup and transition to Secondary state</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="mark.benvenuto@mongodb.com">Mark Benvenuto</assignee>
                                    <reporter username="roger.dawson@ga-digital.com">Roger Dawson</reporter>
                        <labels>
                    </labels>
                <created>Tue, 9 Jun 2015 10:46:30 +0000</created>
                <updated>Fri, 7 Apr 2023 16:14:02 +0000</updated>
                            <resolved>Mon, 3 Aug 2015 20:29:47 +0000</resolved>
                                    <version>3.0.0</version>
                    <version>3.0.1</version>
                    <version>3.0.2</version>
                    <version>3.0.3</version>
                                                    <component>Performance</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="995429" author="ramon.fernandez" created="Mon, 3 Aug 2015 20:29:08 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rogerdawson7&quot; class=&quot;user-hover&quot; rel=&quot;rogerdawson7&quot;&gt;rogerdawson7&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19052&quot; title=&quot;Remove sizeStorer recalculations at startup with WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19052&quot;&gt;&lt;del&gt;SERVER-19052&lt;/del&gt;&lt;/a&gt; should have addressed the root cause of the problem you described in this ticket, so I&apos;m going to mark this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19052&quot; title=&quot;Remove sizeStorer recalculations at startup with WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19052&quot;&gt;&lt;del&gt;SERVER-19052&lt;/del&gt;&lt;/a&gt;. The fix is included in the 3.0.5 release, which &lt;a href=&quot;https://www.mongodb.org/downloads&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;can be downloaded here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="947541" author="mark.benvenuto" created="Mon, 22 Jun 2015 22:47:22 +0000"  >&lt;p&gt;The Azure &quot;Host Cache Preference&quot; recommendation is for the MMapV1 storage engine. It is to workaround to an Azure Storage driver issue specific to memory mapped files. The Azure issue does not affect the WiredTiger storage engine.&lt;/p&gt;

&lt;p&gt;In terms of background flush performance, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-12401&quot; title=&quot;Improve the memory-mapped files flush on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-12401&quot;&gt;&lt;del&gt;SERVER-12401&lt;/del&gt;&lt;/a&gt; does describe the issue. Please note that the issue was written before &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-13681&quot; title=&quot;MongoDB stalls during background flush on Windows&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-13681&quot;&gt;&lt;del&gt;SERVER-13681&lt;/del&gt;&lt;/a&gt; was fixed which mitigated the related performance impact of slow flushes.&lt;/p&gt;

&lt;p&gt;In terms of your disk performance, from the data you gathered I can tell you the following:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;There was an average of 42.455 &lt;tt&gt;Disk Reads/sec&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;There was an average of 25 ms per disk read (i.e., &lt;tt&gt;Avg. Disk sec/Read&lt;/tt&gt; counter)&lt;/li&gt;
	&lt;li&gt;Therefore on average, the system spent 1.061 seconds worth of reads per second. This meant it parallelized reads ever so slightly.&lt;/li&gt;
	&lt;li&gt;This makes sense since we also saw an &lt;tt&gt;Avg. Disk Read Queue Length&lt;/tt&gt; of 1.063.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;We can see from this cursory analysis that the slow start was caused by high latency disk access. We have fixed the underlying cause in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-19052&quot; title=&quot;Remove sizeStorer recalculations at startup with WiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-19052&quot;&gt;&lt;del&gt;SERVER-19052&lt;/del&gt;&lt;/a&gt;.  The process which reads in all the collection info for collections with &amp;lt; 10,000 documents is also executed with no parallelization, which makes this startup job particularly painful in this case.&lt;/p&gt;</comment>
                            <comment id="946731" author="JIRAUSER1272459" created="Mon, 22 Jun 2015 10:37:40 +0000"  >&lt;p&gt;Just for reference, the latest restart took 2 hours 1 minute, suggesting a 20% reduction in times by using an Azure Standard VM instance over a Basic one.&lt;/p&gt;</comment>
                            <comment id="946718" author="JIRAUSER1272459" created="Mon, 22 Jun 2015 09:46:44 +0000"  >&lt;p&gt;Hi Dan, Mark,&lt;/p&gt;

&lt;p&gt;really sorry but I didn&apos;t realise the performance test Dan asked me to perform should have been during the MongoDB service starting up. I&apos;ve repeated the tests while the system is in the starting up state (i.e. where its scanning all the files). The zipped up log file is attached.&lt;/p&gt;

&lt;p&gt;Regarding your questions:&lt;/p&gt;

&lt;p&gt;1) The instance is currently a basic tier. For the most recent performance test (attached) I have changed the instance to a Standard one in order to see if the down time decreases and thus to see how much of an impact the increase in IOPS from 300 to 500 has.&lt;/p&gt;

&lt;p&gt;2) The Azure &apos;Host Cache Preference&apos; for the MongoDB data drive is currently set to read/write as per the Azure section of the production notes (&lt;a href=&quot;http://docs.mongodb.org/manual/administration/production-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/administration/production-notes/&lt;/a&gt;). Could you confirm whether this is the preferred setting for WiredTiger AND MMAP? We have read information regarding this setting and they seem to recommend setting the cache to &apos;None&apos; for database storage.&lt;/p&gt;

&lt;p&gt;3) The documents are generally a few KB&apos;s in size. Most will be small (1-4 KB&apos;s) but there will be 200 or so collections with document sizes as high as 18 KB in them.&lt;/p&gt;

&lt;p&gt;We understand with regards to the background flushing differences between Windows and Linux, as we came across this issue &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-12401&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-12401&lt;/a&gt; a few days ago so things became clearer (maybe worth adding to the MongoDB documentation?).&lt;/p&gt;

&lt;p&gt;One other observation, which might help. We&apos;ve created another Linux VM, but this time using 4 striped disks, and we are seeing no better performance regarding the background flushing (granted this is MMAP so slightly off-topic), but does this suggest perhaps that the disk latencies are negating the benefits of striping?&lt;/p&gt;

&lt;p&gt;Many Thanks, &lt;/p&gt;

&lt;p&gt;Roger&lt;/p&gt;</comment>
                            <comment id="945816" author="mark.benvenuto" created="Fri, 19 Jun 2015 18:10:50 +0000"  >&lt;p&gt;Roger, I examined the log data with Dan. &lt;em&gt;Assuming this snapshot was captured during startup&lt;/em&gt;, we noticed that in the 15 minute snapshot: the system read ~54 MB, and wrote ~533 MB of data. Neither of these are large values, and it does not look like you are I/O bound. In terms of disk latency, we saw average times of 20ms which is not great, but this is probably normal for Azure Storage.&lt;/p&gt;

&lt;p&gt;A few more questions about your setup:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;You say above you are using an A3, is this from the standard or basic tier?&lt;/li&gt;
	&lt;li&gt;Do the disks have any caching enabled? (None, Read-only or Read/Write)&lt;/li&gt;
	&lt;li&gt;In these collections with &amp;lt; 10,000 documents, what is the average size of these documents? a few kbs, a few mbs?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;In terms of background flush performance, the Linux OS issues asynchronous I/Os during flushes of memory mapped files while the Windows OS issues synchronous I/Os. It is not a difference in MongoDB, but the OS.&lt;/p&gt;</comment>
                            <comment id="945552" author="dan@10gen.com" created="Fri, 19 Jun 2015 15:03:01 +0000"  >&lt;p&gt;I&apos;ve created a ticket to track removing the sizeStorer update work at startup.&lt;/p&gt;</comment>
                            <comment id="945415" author="dan@10gen.com" created="Fri, 19 Jun 2015 13:18:12 +0000"  >&lt;p&gt;Mark, can you help me analyze this?&lt;/p&gt;</comment>
                            <comment id="945262" author="JIRAUSER1272459" created="Fri, 19 Jun 2015 09:55:01 +0000"  >&lt;p&gt;Hello Dan,&lt;/p&gt;

&lt;p&gt;here is the log output as requested,&lt;/p&gt;

&lt;p&gt;Many Thanks,&lt;/p&gt;

&lt;p&gt;Roger&lt;/p&gt;</comment>
                            <comment id="944350" author="JIRAUSER1272459" created="Thu, 18 Jun 2015 16:25:15 +0000"  >&lt;p&gt;Hi Dan,&lt;/p&gt;

&lt;p&gt;unfortunately the Linux VM we have created is running using MMAP. We might be spinning up new Linux VM&apos;s though to replace our Windows ones so we could configure one for WiredTiger. What we have noticed is considerably better performance from the Linux VM MMAP than the Windows MMAP VM regarding background flushing, although we can&apos;t understand why as the Linux VM is an A2 instance with a single disk (500 IOPS) and the Windows VM is an A3 instance with 4 striped disks (500 IOPS each). Windows ~ 20 seconds whereas Linux ~4 seconds. &lt;/p&gt;

&lt;p&gt;I&apos;ll run the command you mention when I get a minute tomorrow morning and get the results back to you,&lt;/p&gt;

&lt;p&gt;Thanks, &lt;/p&gt;

&lt;p&gt;Roger&lt;/p&gt;</comment>
                            <comment id="944265" author="dan@10gen.com" created="Thu, 18 Jun 2015 15:27:14 +0000"  >&lt;p&gt;Thanks for sharing that info Roger.  The startup time is dominated by MongoDB scanning your collections that are less than 10,000 documents.  MongoDB scans these collections at startup to verify that the count and size statistics are correct for small collections.  Under normal circumstances, this should be very fast.  I set up server on linux with 1200 collections and document counts matching your breakdown above, using small documents of around 50 bytes. The startup time on my linux server with SSD&apos;s is around 4 seconds.  The size of the documents could be a factor, but it could also be something to do with windows or azure storage.&lt;/p&gt;

&lt;p&gt;I&apos;m seeing in the logfile you attached that a single scan is taking several seconds, up to around 30 seconds which really doesn&apos;t make sense to me:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-06-17T13:34:01.015+0000 D STORAGE  [initandlisten] doing scan of collection OurCustomerDBV3.RC_PPUOVR_21_375630 to get info&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2015-06-17T13:34:35.906+0000 D STORAGE  [initandlisten] looking up metadata for: OurCustomerDBV3.RC_PPUOVR_21_375701 @ RecordId(276)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Have you seen the same issue on your linux VM?&lt;/p&gt;

&lt;p&gt;If you have time, it would be helpful to see a performance counter trace to understand why it is running so slow. You will need to run the following commands, and it will collect a trace file for 15 minutes of 1 second samples, and then stop. After it stops, attach the information to the ticket so understand how much write traffic there is and if is particularly high disk latency from your storage system in the first place.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;logman delete mongodb_log&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;logman create counter mongodb_log -c &quot;\Processor(_Total)\% Processor Time&quot; &quot;\Processor(_Total)\% Privileged Time&quot; &quot;\PhysicalDisk(_Total)\*&quot; -f bin -rf 00:15:00 -si 1 -ow&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;logman start mongodb_log&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;later run&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;logman query mongodb_log to find the file(s), zip and attach to jira&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="943872" author="JIRAUSER1272459" created="Thu, 18 Jun 2015 07:25:55 +0000"  >&lt;p&gt;Hi Dan,&lt;/p&gt;

&lt;p&gt;here is the very rough breakdown. We have approximately 150-200 of each of the collections I am about to describe:&lt;/p&gt;

&lt;p&gt;Capped collection ~ 3000 docs&lt;br/&gt;
Large overview docs ~ 3000 docs max, 2500 docs average&lt;br/&gt;
Raw data 1 ~ 300,000 docs max, 200,000 docs average&lt;br/&gt;
Raw data 2 ~ 3,000,000 docs max, 2,200,000 docs average&lt;br/&gt;
New raw data 1 ~ 3000 docs&lt;br/&gt;
New raw data 2 ~ 50,000 docs max, 30,000 docs average&lt;/p&gt;

&lt;p&gt;The two new raw data collections have been created in to improve write performance as the &apos;raw data 2&apos; collections are approaching 1.4 GB in size and 300 MB index sizes.&lt;/p&gt;

&lt;p&gt;So to answer your question, I would say of the 1200 or so collections we have about 50% are less than 10,000 documents in size.&lt;/p&gt;

&lt;p&gt;Thanks, Roger&lt;/p&gt;</comment>
                            <comment id="943151" author="dan@10gen.com" created="Wed, 17 Jun 2015 16:57:23 +0000"  >&lt;p&gt;I think I know what the problem is.  Would it be possible to say how many of the collections you have contain less than 10,000 documents?&lt;/p&gt;</comment>
                            <comment id="943029" author="JIRAUSER1272459" created="Wed, 17 Jun 2015 15:34:58 +0000"  >&lt;p&gt;Hi Dan,&lt;/p&gt;

&lt;p&gt;I&apos;ve attached the directory listings for the data directory and journal directory following what I believe was a slow, but clean shutdown. I carried out the shutdown using the db.shutdownServer command on the Wired Tiger secondary as this seems the only reliable way to get no errors on shutdown. You&apos;ll see I also attached the log output from the node as it was shutdown and then restarted. It took approximately 2.5 hours to get up and running again. I&apos;m guessing the extra 30 minutes is because we have more collections now. We started logging to new collections as some were very large and we thought might be responsible for the performance issues.&lt;/p&gt;

&lt;p&gt;Thanks, Roger&lt;/p&gt;</comment>
                            <comment id="942768" author="JIRAUSER1272459" created="Wed, 17 Jun 2015 11:34:13 +0000"  >&lt;p&gt;Hi Dan, okay, I&apos;ll do that but it might take a while. We have add a Linux VM to the replica set as we want to see if it is any more performant/reliable than Windows and it&apos;s still replicating at the moment (building the indexes for the last 24 hours so hopefully on the last leg). Once this is completed we can risk taking the wired tiger node out of the replica set. We are down to a primary (mmap), secondary (WT) and arbiter as one of our MMAP nodes restarted following Windows updates and irrecoverably fails on startup (it&apos;s happened twice now) and the Linux box currently replicating has replaced one of the wired tiger nodes. &lt;/p&gt;

&lt;p&gt;I&apos;m not sure if its related but we&apos;re experiencing a lot of issues with the whole replica set these days such as nodes failing to start following unclean shutdowns (which seem to occur almost always but its only the mmap&apos;s that have actually crashed irrecoverably), very high background flush times etc. We&apos;ve tried a lot of things to improve performance and it has made our processing considerably quicker but none of it seems to be helping the startup times, high background flush times etc. Most of these issues seemed to begin occurring when we transitioned to 3.0 but that could be a coincidence. We wanted to run version 3.0 as we found we needed very large page files sizes (~50GB at the moment) and though WiredTiger would solve this as well as be faster.&lt;/p&gt;</comment>
                            <comment id="942748" author="dan@10gen.com" created="Wed, 17 Jun 2015 10:51:16 +0000"  >&lt;p&gt;Hi Roger, I appreciate your help here.  Sorry it wasn&apos;t clear in the last request, but in addition to the dbpath contents, we also need the contents of the journal directory, but after a clean shutdown and before restarting the server so that we can verify that the storage engine is not attempting to recover.&lt;/p&gt;

&lt;p&gt;Also, when you do restart your mongod, can you start with verbose logging and attach the output?  &lt;tt&gt;mongod --storageEngine wiredTiger -v&lt;/tt&gt; from the command line or in the config file:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;systemLog:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;   verbosity: 1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Dan&lt;/p&gt;</comment>
                            <comment id="942734" author="JIRAUSER1272459" created="Wed, 17 Jun 2015 10:26:31 +0000"  >&lt;p&gt;Directory contents of our wired tiger secondary as requested.&lt;/p&gt;</comment>
                            <comment id="941457" author="JIRAUSER1272459" created="Tue, 16 Jun 2015 07:19:31 +0000"  >&lt;p&gt;Hi Ramon,&lt;/p&gt;

&lt;p&gt;I&apos;ll try and get the directory listing you requested. Yes, we have had quite a few unclean shutdowns on this machine. Since we moved to version 3.0 we&apos;ve struggled to have a clean shutdown on any of our machines. However, I did manage what I believe was a clean shutdown recently, as indicated by the log above, when I used the db.shutdownServer command. However, this still resulted in the system taking 2 hours to startup, as illustrated by the last two log entries.&lt;/p&gt;

&lt;p&gt;Thanks, &lt;/p&gt;

&lt;p&gt;Roger&lt;/p&gt;</comment>
                            <comment id="941191" author="ramon.fernandez" created="Mon, 15 Jun 2015 22:03:15 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rogerdawson7&quot; class=&quot;user-hover&quot; rel=&quot;rogerdawson7&quot;&gt;rogerdawson7&lt;/a&gt;, can you please send us a directory listing of your dbpath? Also, have you had any unclean shutdowns on this machine? Other than a slow storage layer (WT needs to open over 2K files on startup in your case) the other thing that may cause this behavior is a recovery operation, which may happen after an unclean shutdown.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="935205" author="JIRAUSER1272459" created="Tue, 9 Jun 2015 15:34:33 +0000"  >&lt;p&gt;Just to add to this, it doesn&apos;t look to be limited to unclean shutdowns. I had to shut my Wired Tiger secondary down so I logged into MongoDB (via shell) and executed db.shutdownServer with a 120 second timeout in an attempt to shutdown gracefully following comments I&apos;ve seen in the forums. The operation finished after a few seconds and didn&apos;t appear to have any issues. Upon restart my server still took a couple of hours to transition to the secondary state. The log output is shown below, please note the time difference between the last two log entries:&lt;/p&gt;

&lt;p&gt;15-06-09T13:26:32.981+0000 I COMMAND  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn4501&amp;#93;&lt;/span&gt; terminating, shutdown command received&lt;br/&gt;
2015-06-09T13:26:33.233+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn4512&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
2015-06-09T13:26:33.359+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;WiredTigerRecordStoreThread for local.oplog.rs&amp;#93;&lt;/span&gt; shutting down&lt;br/&gt;
2015-06-09T13:26:33.462+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn4513&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
2015-06-09T13:26:33.887+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn4510&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
2015-06-09T13:26:34.580+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn4413&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
2015-06-09T13:26:34.779+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3125&amp;#93;&lt;/span&gt; got request after shutdown()&lt;br/&gt;
2015-06-09T13:26:37.237+0000 W NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplExecNetThread-107&amp;#93;&lt;/span&gt; Failed to connect to 10.0.1.70:27017 after 5000 milliseconds, giving up.&lt;br/&gt;
2015-06-09T13:26:37.237+0000 I REPL     &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; Stopping replication applier threads&lt;br/&gt;
2015-06-09T13:26:37.239+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; now exiting&lt;br/&gt;
2015-06-09T13:26:37.239+0000 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: going to close listening sockets...&lt;br/&gt;
2015-06-09T13:26:37.239+0000 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; closing listening socket: 308&lt;br/&gt;
2015-06-09T13:26:37.240+0000 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: going to flush diaglog...&lt;br/&gt;
2015-06-09T13:26:37.240+0000 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: going to close sockets...&lt;br/&gt;
2015-06-09T13:26:37.266+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; WiredTigerKVEngine shutting down&lt;br/&gt;
2015-06-09T13:26:47.991+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Service Stop is waiting for storage engine to finish shutdown&lt;br/&gt;
2015-06-09T13:26:54.694+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: removing fs lock...&lt;br/&gt;
2015-06-09T13:26:54.700+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; dbexit:  rc: 49&lt;br/&gt;
2015-06-09T13:34:16.290+0000 I CONTROL  ***** SERVER RESTARTED *****&lt;br/&gt;
2015-06-09T13:34:16.885+0000 I CONTROL  Trying to start Windows service &apos;MongoDB&apos;&lt;br/&gt;
2015-06-09T13:34:16.896+0000 I STORAGE  Service running&lt;br/&gt;
2015-06-09T13:34:17.235+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),&lt;br/&gt;
2015-06-09T15:23:02.179+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Starting WiredTigerRecordStoreThread local.oplog.rs&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="211844">SERVER-19052</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="212347">DOCS-5699</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="78326" name="datadircontents.txt" size="298482" author="rogerdawson7" created="Wed, 17 Jun 2015 15:34:57 +0000"/>
                            <attachment id="78242" name="dircontents.txt" size="298482" author="rogerdawson7" created="Wed, 17 Jun 2015 10:26:31 +0000"/>
                            <attachment id="78327" name="journaldircontents.txt" size="432" author="rogerdawson7" created="Wed, 17 Jun 2015 15:34:57 +0000"/>
                            <attachment id="78654" name="mongodb_log_000001.zip" size="227258" author="rogerdawson7" created="Fri, 19 Jun 2015 09:55:01 +0000"/>
                            <attachment id="78837" name="mongodb_log_000002.zip" size="250595" author="rogerdawson7" created="Mon, 22 Jun 2015 09:46:44 +0000"/>
                            <attachment id="78328" name="mongowtstartup.log" size="3023203" author="rogerdawson7" created="Wed, 17 Jun 2015 15:34:58 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 9 Jun 2015 16:12:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 28 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 28 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[Windows]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.benvenuto@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>roger.dawson@ga-digital.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl3pj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsb8lb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="632">Platform 7 08/10/15</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Restart the Windows Server MongoDB is running on. On restart the system will take about 2 hours to get to the &apos;Secondary&apos; state during which times it appears to be scanning every collection and index file (~2000 for us).&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsfydz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>