<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:47:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-38089] Fatal error 40292 due to missing oplog entry at the start time stamp</title>
                <link>https://jira.mongodb.org/browse/SERVER-38089</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;A series of weird happenings have ended up in a situation triggering Fatal Error 40292 in&#160;&lt;tt&gt;mongo/db/repl/replication_recovery.cpp&lt;/tt&gt; line 133.&lt;/p&gt;

&lt;p&gt;We have a PSA configuration. The entire issue started when we our DB approached (or hit) the disk limit. We resized the disk about mid October. From there on we&apos;ve not been able to have a stable system. The servers keep going down.&lt;/p&gt;

&lt;p&gt;I got involved two days ago, so I don&apos;t have the entire history.&lt;/p&gt;

&lt;p&gt;I found that our system was triggering &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4335&quot; title=&quot;Don&amp;#39;t fail rollback_to_stable due to sweep activity&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4335&quot;&gt;&lt;del&gt;WT-4335&lt;/del&gt;&lt;/a&gt;, with version 4.0.1. I upgraded to 4.0.4 yesterday. I am guessing now the fatal error is triggering at the root cause. (The &lt;a href=&quot;https://jira.mongodb.org/browse/WT-4335&quot; title=&quot;Don&amp;#39;t fail rollback_to_stable due to sweep activity&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-4335&quot;&gt;&lt;del&gt;WT-4335&lt;/del&gt;&lt;/a&gt; was triggering after all the oplog replay, now the error is at the start itself.)&lt;/p&gt;


&lt;p&gt;&lt;tt&gt;2018-11-12T05:41:22.612+0000 I FTDC &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Initializing full-time diagnostic data capture with directory &apos;/mongo/rs0_8/diagnostic.data&apos;&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-12T05:41:22.614+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Rollback ID is 12&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-12T05:41:22.614+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Recovering from stable timestamp: Timestamp(1539794142, 116) (top of oplog: { ts: Timestamp(1541830881, 1), t: 85 }, appliedThrough: { ts: Timestamp(0, 0), t: -1 }, TruncateAfter: Timestamp(0, 0))&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-12T05:41:22.614+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Starting recovery oplog application at the stable timestamp: Timestamp(1539794142, 116)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-12T05:41:22.614+0000 I REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Replaying stored operations from { : Timestamp(1539794142, 116) } (exclusive) to { : Timestamp(1541830881, 1) } (inclusive).&lt;/tt&gt;&lt;br/&gt;
{{2018-11-12T05:41:22.617+0000 F REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Oplog entry at { : Timestamp(1539794142, 116) } is missing; actual entry found is { : Timestamp(1539794616, 312) }}}&lt;br/&gt;
&lt;tt&gt;2018-11-12T05:41:22.617+0000 F - &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Fatal Assertion 40292 at src/mongo/db/repl/replication_recovery.cpp 134&lt;/tt&gt;&lt;br/&gt;
{{2018-11-12T05:41:22.617+0000 F - &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; }}&lt;/p&gt;

&lt;p&gt;For reference, here is log with 4.0.1 (prior to the above error).&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;2018-11-11T10:23:54.182+0000 I ROLLBACK &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; Rolling back to the stable timestamp. StableTimestamp: Timestamp(1539794142, 116) Initial Data Timestamp: Timestamp(1539794142, 116)&lt;/tt&gt;&lt;tt&gt;2018-11-11T10:23:54.275+0000 E STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; WiredTiger error (22) &lt;span class=&quot;error&quot;&gt;&amp;#91;1541931834:274955&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;23314:0x7f35f5bd7700&amp;#93;&lt;/span&gt;, WT_CONNECTION.rollback_to_stable: rollback_to_stable illegal with active transactions: Invalid argument Raw: &lt;span class=&quot;error&quot;&gt;&amp;#91;1541931834:274955&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;23314:0x7f35f5bd7700&amp;#93;&lt;/span&gt;, WT_CONNECTION.rollback_to_stable: rollback_to_stable illegal with active transactions: Invalid argument&lt;/tt&gt;&lt;tt&gt;2018-11-11T10:23:54.275+0000 F ROLLBACK &lt;span class=&quot;error&quot;&gt;&amp;#91;rsBackgroundSync&amp;#93;&lt;/span&gt; RecoverToStableTimestamp failed. :: caused by :: UnrecoverableRollbackError: Error rolling back to stable. Err: Invalid argument&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="632481">SERVER-38089</key>
            <summary>Fatal error 40292 due to missing oplog entry at the start time stamp</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="DrArunL">Arun Lakhotia [X]</reporter>
                        <labels>
                            <label>SWNA</label>
                    </labels>
                <created>Mon, 12 Nov 2018 16:03:15 +0000</created>
                <updated>Wed, 12 Dec 2018 19:19:06 +0000</updated>
                            <resolved>Wed, 12 Dec 2018 19:19:06 +0000</resolved>
                                    <version>4.0.1</version>
                    <version>4.0.4</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="2088792" author="daniel.hatcher" created="Wed, 12 Dec 2018 19:18:33 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;I&apos;m sorry that you are still encountering some inconsistent data. I can recommend our &lt;a href=&quot;https://docs.mongodb.org/manual/support&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Technical Support team&lt;/a&gt; for future issues with Automation (and data recovery) as they are experts on those subjects.&lt;/p&gt;

&lt;p&gt;Please let us know if you discover anything else that could help identify the root cause.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2085982" author="drarunl" created="Mon, 10 Dec 2018 19:14:59 +0000"  >&lt;p&gt;Daniel and the rest of the team:&lt;/p&gt;

&lt;p&gt;Thanks for your responses to this Issue, and providing helpful guidance. Yes, at this time there is nothing more we can do on this issue. At best, I (and I am sure others) can benefit from some Do&apos;s and Don&apos;ts. This could certainly be very useful for people who also use the mongodb web service for automation and monitoring.&lt;/p&gt;

&lt;p&gt;It so happens I am still finding inconsistent state in the DB. But its not something you can help me with.&lt;/p&gt;

&lt;p&gt;So you may go ahead and close the issue.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2085651" author="daniel.hatcher" created="Mon, 10 Dec 2018 16:36:56 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;I&apos;ve been trying to think of any alternative possibilities but I believe that your assessment is accurate. MongoDB has known issues when it actually runs out of disk space so that situation plus multiple &lt;tt&gt;mongod&lt;/tt&gt;&apos;s trying to access the same data would absolutely cause problems. For the former issue, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-13811&quot; title=&quot;Deal better/Fail more gracefully when mongoD runs out of disk space&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-13811&quot;&gt;SERVER-13811&lt;/a&gt; would track the work when we come up with a viable solution.&lt;/p&gt;

&lt;p&gt;As there are so many unknowns as to the original cause of the issue and your system is now back up and running, I think this ticket can be closed. Do you have anything else for us?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2079149" author="drarunl" created="Mon, 3 Dec 2018 21:19:14 +0000"  >&lt;p&gt;Hi Danny:&lt;/p&gt;

&lt;p&gt;I think I&apos;d need a therapist to make him talk &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;, he may be scared of reliving the experience or he is scared he will be held responsible. I have been very clear all along that none of this will be held against him. And that I want to know what may have happened so we do not repeat it.&lt;/p&gt;

&lt;p&gt;FWIW, my initial upload of log files contained logs from just before, during, and after. So there is a possibility the log contains some useful story.&lt;/p&gt;

&lt;p&gt;Here are steps that I am aware of:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Early Oct&apos;18: We have a 4T x 2 replica set. It was running close to full. I don&apos;t know if the system had started giving errors/warnings. Per my colleague, he observed we were running out of space.&lt;/li&gt;
	&lt;li&gt;Mid-October: He manually reconfigured the disks to increase size to 4T; and brought the servers up manually. He didn&apos;t know about the automation agent.&lt;/li&gt;
	&lt;li&gt;After some period of operation, the system (as observed from external operation) would shut down. And he&apos;d start the server again. This kept going for some time.&lt;/li&gt;
	&lt;li&gt;I got involved a day or two before I started this note. Which means the system had been flaky for over 3 weeks.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The possibilities I could think of:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;We were already out of disk space, or close, before the resizing. Some operations would succeed, others fail, leading to inconsistent state. So the damage was already done. There could be some issue with how mongodb performs when it hits disk limits.&lt;/li&gt;
	&lt;li&gt;The resizing of disk on Google Cloud Platform has some issues that messed up with the data. (Low probability, based on all other indications.)&lt;/li&gt;
	&lt;li&gt;There was some contention in a &quot;root mongodb&quot; and &quot;non-root mongodb&quot;, and they didn&apos;t know how to resolve the contention, especially when they may get lock alternately (because a process is killed or dies).&lt;/li&gt;
	&lt;li&gt;A combination of #1 and #3.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2079101" author="daniel.hatcher" created="Mon, 3 Dec 2018 20:52:53 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;I think your explanation makes sense as a possible original cause for these issues. Would you happen to have the list of steps the other user followed back in October? If we can identify exactly what happened then it may reveal something that we can fix.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2076954" author="drarunl" created="Fri, 30 Nov 2018 17:11:00 +0000"  >&lt;p&gt;Based on the above statement that automation may have interfered with recovery, may be this may help identify the root cause of where this all started.&lt;/p&gt;

&lt;p&gt;When we reconfigured our disk (about October 17), and restarted mongo, the person responsible was not aware of the automation agent. He started mongod manually AND he started it as &lt;tt&gt;root&lt;/tt&gt;. As he was configuring system, there was automation agent running in the background competing with the same work.&#160;&lt;/p&gt;

&lt;p&gt;He wasn&apos;t aware of the contention. He would look at the logs, see issues related to file ownership, and change ownerships to &lt;tt&gt;root&lt;/tt&gt;. But the process itself was running as&#160;&lt;tt&gt;mongodb&lt;/tt&gt; user.&lt;/p&gt;

&lt;p&gt;Maybe this fight between two mongodb processes (and more on replica set), put the system in some state that it usually doesn&apos;t enter. Add to it the file limit, and the issue ballooned.&lt;/p&gt;

&lt;p&gt;So my plan is that once we reach a stable operation manually, we will then configure automation agent. And from there on we will be very careful when doing anything manually.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2076861" author="daniel.hatcher" created="Fri, 30 Nov 2018 16:25:44 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Do you have any dos/don&apos;ts for using the automation agent. At this time, I have cut it out entirely. I can see the need for using automation agent when configuring or reconfiguring machines. I don&apos;t see if it serves a lot of purpose when a system is in steady state. (May be it does, and I&apos;d like to learn.)&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Actually, I think Automation is very useful for the normal running state of systems. If a &lt;tt&gt;mongod&lt;/tt&gt; process managed by Automation goes down for any reason, the Automation agent will attempt to bring it back up. In situations where the node won&apos;t start back up there&apos;s no real advantage but otherwise you don&apos;t have to worry about a &lt;tt&gt;mongod&lt;/tt&gt; silently crashing and never restarting. It also makes things like version upgrades easier and as you mentioned other reconfigurations.&lt;/p&gt;

&lt;p&gt;The downside of Automation is that it is &quot;optimistic&quot;. It assumes that any changes that occur are happening to a relatively healthy system. In this scenario, we have been attempting some configurations that you would never apply to a healthy system so Automation tries and fails to fix the problems it sees. Unfortunately, some of those attempted fixes directly contradict what we are telling the system to do. Thus, for periods of abnormal maintenance I believe it is beneficial to remove the relevant processes from Automation but in all other circumstances to take advantage of its features.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;ulimit. See below. I see that the open files in my settings is much smaller (1024) compared to what is on the mongodb page (21000). &lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;We actually recommend &lt;a href=&quot;https://docs.mongodb.com/manual/reference/ulimit/index.html#recommended-ulimit-settings&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;a minimum value of 64000 for the &quot;max open files&quot; ulimit&lt;/a&gt;. Of course you can go higher if you believe your system requires it. It may not have impacted you before due to a relatively low amount of load on MongoDB under most circumstances but it is useful to have the higher value just in case.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;You didn&apos;t mention, but related. How do monitoring agent and backup agent interfere with mongo operation. I like to use at least the monitoring agent. I can do without backup agent, for we are using disk snapshots as backups.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;The impact of Monitoring should be so negligible that it is not even noticed and Backup should only be slightly more. However, if you have no intention of ever using the Backup feature I see no reason to install the agent.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The attack happened because the number of open files exceeded the ulimit settings. Does that by any chance point to some issue in the server code? Is there a possibility that files are opened, and not being closed?&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Possibly, but I do not see any indication of that. 1024 really is a low value so it is not surprising you ran into that issue. If you do run out of open files at a much larger value, like the recommended 64000, then that would be something to investigate.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2076848" author="drarunl" created="Fri, 30 Nov 2018 16:17:49 +0000"  >&lt;p&gt;About the PANIC attack.&lt;/p&gt;

&lt;p&gt;The attack happened because the number of open files exceeded the ulimit settings. Does that by any chance point to some issue in the server code? Is there a possibility that files are opened, and not being closed?&lt;/p&gt;

&lt;p&gt;I don&apos;t per se need an explanation. I am sure there are very good design reasons why one may open such large number of files. So this is just a comment, in case there is some issue that may have been overlooked.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2076806" author="drarunl" created="Fri, 30 Nov 2018 15:49:30 +0000"  >&lt;p&gt;Danny:&lt;/p&gt;

&lt;p&gt;My status: I am currently running standalone configuration.&#160;In this mode we did find some corruption in the data, that is, our collections were out of sync. That is because we never focused on robustness in the event of such outages.&#160;&lt;/p&gt;

&lt;p&gt;At this time, I am more concerned about getting back into an operational state using RS. Since I do not really know what caused the issue, I am being ultra cautious.&#160;&lt;/p&gt;

&lt;p&gt;I have understood the following from your note;&lt;/p&gt;

&lt;p&gt;1) Automation agent has to be used with care.&lt;/p&gt;

&lt;p&gt;Do you have any dos/don&apos;ts for using the automation agent. At this time, I have cut it out entirely. I can see the need for using automation agent when configuring or reconfiguring machines. I don&apos;t see if it serves a lot of purpose when a system is in steady state. (May be it does, and I&apos;d like to learn.)&lt;/p&gt;

&lt;p&gt;2) ulimit. See below. I see that the open files in my settings is much smaller (1024) compared to what is on the mongodb page (21000).&#160;&lt;/p&gt;

&lt;p&gt;&#160;I can see that as the first tangible cause for the PANIC attack. So this is great.&lt;/p&gt;

&lt;p&gt;3) You didn&apos;t mention, but related. How do monitoring agent and backup agent interfere with mongo operation. I like to use at least the monitoring agent. I can do without backup agent, for we are using disk snapshots as backups.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;For reference, here is my ulimit settings&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ubuntu@mongod-0:~$ ulimit -a&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;core file size          (blocks, -c) 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;data seg size           (kbytes, -d) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;scheduling priority             (-e) 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;file size               (blocks, -f) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;pending signals                 (-i) 483382&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;max locked memory       (kbytes, -l) 64&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;max memory size         (kbytes, -m) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;open files                      (-n) 1024&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;pipe size            (512 bytes, -p) 8&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;POSIX message queues     (bytes, -q) 819200&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;real-time priority              (-r) 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;stack size              (kbytes, -s) 8192&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;cpu time               (seconds, -t) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;max user processes              (-u) 483382&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;virtual memory          (kbytes, -v) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;file locks                      (-x) unlimited&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;And limits of the mongod process on the current functional machine. We have not changed any configuration here, so I am sure that the processes were running with same limits earlier.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ubuntu@mongod-0:~$ cat /proc/11087/limits&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Limit                     Soft Limit           Hard Limit           Units     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max cpu time              unlimited            unlimited            seconds   &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max file size             unlimited            unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max data size             unlimited            unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max stack size            8388608              unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max core file size        0                    unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max resident set          unlimited            unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max processes             483382               483382               processes &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max open files            1024                 1048576              files     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max locked memory         65536                65536                bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max address space         unlimited            unlimited            bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max file locks            unlimited            unlimited            locks     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max pending signals       483382               483382               signals   &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max msgqueue size         819200               819200               bytes     &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max nice priority         0                    0                    &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max realtime priority     0                    0                    &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Max realtime timeout      unlimited            unlimited            us  &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2071731" author="daniel.hatcher" created="Mon, 26 Nov 2018 18:33:02 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;Thank you for providing the extended amount of detail surrounding your environment. I think there&apos;s enough evidence that Automation has been interfering with your recovery efforts so I am glad to see that you are currently running things manually until everything is healthy again.&lt;/p&gt;

&lt;p&gt;In regards to the latest panic during the repair, it appears that we hit the limit of open files on the Linux server. Do you happen to know the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/ulimit/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;ulimit settings&lt;/a&gt; these servers are using? If you have a &lt;tt&gt;mongod&lt;/tt&gt; currently up and running on one of the problem servers, the following is probably the best way to obtain that information:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;cat /proc/$pid/limits&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;blockquote&gt;
&lt;p&gt;On starting mongod-1 we got an error that it was improperly shutdown. So we started it with --repair.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I can see that the fatal assertion here was the same one that caused you to open this case. I know that there have been multiple copies of data files. Are the data files under this node considered the &quot;clean&quot; ones or are they the known &quot;broken&quot; ones? Do you still have a &quot;clean&quot; copy of the data files that do not cause errors when starting a server against it?&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T20:40:10.026+0000 I REPL     [initandlisten] Recovering from stable timestamp: Timestamp(1539794142, 116) (top of oplog: { ts: Timestamp(1542212444, 1), t: 217 }, appliedThrough: { ts: Timestamp(0, 0), t: -1 }, TruncateAfter: Timestamp(0, 0))&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T20:40:10.026+0000 I REPL     [initandlisten] Starting recovery oplog application at the stable timestamp: Timestamp(1539794142, 116)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T20:40:10.026+0000 I REPL     [initandlisten] Replaying stored operations from { : Timestamp(1539794142, 116) } (exclusive) to { : Timestamp(1542212444, 1) } (inclusive).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T20:40:10.029+0000 F REPL     [initandlisten] Oplog entry at { : Timestamp(1539794142, 116) } is missing; actual entry found is { : Timestamp(1539795252, 332) }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T20:40:10.029+0000 F -        [initandlisten] Fatal Assertion 40292 at src/mongo/db/repl/replication_recovery.cpp 134&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;As far as I can see, each iteration that has been tested thus far has been PSA or a standalone node. Have there been any attempts to run the data as a single-member replica set or as a member of a PSS replica set? The latter would involve your arbiter server hosting actual data load so I am not sure if this is too onerous.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2070139" author="drarunl" created="Thu, 22 Nov 2018 19:10:45 +0000"  >&lt;p&gt;Here is another crash report. Now I have my production system running as standalone. Am using replicaset for experiments. The planned set will have three machines: &lt;tt&gt;mongod-1&lt;/tt&gt;, &lt;tt&gt;mongod-2&lt;/tt&gt;, and &lt;tt&gt;mongod-arbiter&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;On starting &lt;tt&gt;mongod-1&lt;/tt&gt; we got an error that it was improperly shutdown. So we started it with &lt;tt&gt;--repair&lt;/tt&gt;. It crashed after about 35 hours.&#160;&lt;/p&gt;

&lt;p&gt;Here is extract from error triggering the panic.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.338+0000 E STORAGE  [thread1] WiredTiger error (24) [1542878933:337974][17240:0x7fbffff42700], log-server: __directory_list_worker, 48: /mongo/rs0_8//journal: directory-list: opendir: Too many open files Raw: [1542878933:337974][17240:0x7fbffff42700], log-server: __directory_list_worker, 48: /mongo/rs0_8//journal: directory-list: opendir: Too many open files&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.338+0000 E STORAGE  [thread1] WiredTiger error (24) [1542878933:338216][17240:0x7fbffff42700], log-server: __log_prealloc_once, 469: log pre-alloc server error: Too many open files Raw: [1542878933:338216][17240:0x7fbffff42700], log-server: __log_prealloc_once, 469: log pre-alloc server error: Too many open files&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.338+0000 E STORAGE  [thread1] WiredTiger error (24) [1542878933:338232][17240:0x7fbffff42700], log-server: __log_server, 1015: log server error: Too many open files Raw: [1542878933:338232][17240:0x7fbffff42700], log-server: __log_server, 1015: log server error: Too many open files&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.338+0000 E STORAGE  [thread1] WiredTiger error (-31804) [1542878933:338248][17240:0x7fbffff42700], log-server: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1542878933:338248][17240:0x7fbffff42700], log-server: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.384+0000 F -        [WTJournalFlusher] Invariant failure: _waitUntilDurableSession-&amp;gt;log_flush(_waitUntilDurableSession, &quot;sync=on&quot;) resulted in status UnknownError: -31804: WT_PANIC: WiredTiger library panic at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 296&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-22T09:28:53.384+0000 F -        [WTJournalFlusher] ***aborting after invariant() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;I have uploaded all the data. The server was run manually, so you can ignore the automation ogs.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;725d5af2ade5541c3a1411d08aafec1e7c3e4882  mongod-1-2018-11-22T19-03-39.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2067438" author="drarunl" created="Tue, 20 Nov 2018 05:30:44 +0000"  >&lt;p&gt;FYI &#8211; I am running my production server on a standalone mode, which gives me reduced performance.&#160; This is not ideal, but it is better than being completely shut.&lt;/p&gt;

&lt;p&gt;To help with debugging this issue, I am keeping the replicas intact. If you&apos;d like me to try any new builds, turn on more intrusive logging, or any other experiment, I&apos;d be happy to give it a try.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2067433" author="drarunl" created="Tue, 20 Nov 2018 05:08:51 +0000"  >&lt;p&gt;Pardon me for this big blast. It appears there is some mess up with automation going on as well The automation log show lots of permission issues on both sides.&lt;/p&gt;

&lt;p&gt;Also, I found that &lt;tt&gt;mongod-2&lt;/tt&gt; started on an empty database, not on a copy of the old DB. (There was an error in the directory path of the DB configuration which led &lt;tt&gt;mongod&lt;/tt&gt;&#160;to create an empty DB. That explains why it started right away, instead of taking hours.)&lt;/p&gt;

&lt;p&gt;FWIW, this may offer some clues as well.&lt;/p&gt;

&lt;p&gt;In the morning I will have the three servers started afresh without any automation agent.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2067417" author="drarunl" created="Tue, 20 Nov 2018 04:23:47 +0000"  >&lt;p&gt;Continuing from previous, I figure I should give as much data as I can think of. Here is the &lt;tt&gt;local&lt;/tt&gt; directory on &lt;tt&gt;mongod-2&lt;/tt&gt;.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:~# ls -lh  /mongo/rs0_6/local&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;total 136K&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 collection-0-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup 4.0K Nov 20 03:00 collection-2-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 collection-4-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 collection-6-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 collection-8-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 index-1-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup 4.0K Nov 20 03:00 index-3-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 index-5-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 index-7-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup  16K Nov 20 03:01 index-9-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:~# &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;And their SHA1&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:/mongo/rs0_6/local# find . -type f -exec sha1sum {} \;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;5bd422b5f3e0bce89c1e56327c6f05cf2b125cee  ./collection-0-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3a974266d660c996aef5327b8b7e8b69fbdaf7f6  ./index-1-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ef338c4b8d25224f30566e1ade30a1ca9eb26866  ./collection-2-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ef338c4b8d25224f30566e1ade30a1ca9eb26866  ./index-3-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;fd2592fd13660ad11bca8f65579cdd0de5de001e  ./collection-4-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d30d746858bde08873e4ce04d61877811b3ba5f4  ./index-5-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3c985315d6cb096020609164ee53188c0e22e52d  ./collection-6-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;41129285edc0117139d365c0b63313afa0a634d8  ./index-7-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d5c669006a44bfa50cbd42763ddc3a2aa8c02782  ./collection-8-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;82a25b11b9c412f69d810fda82f4563cf8b0f176  ./index-9-714744392904966981.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Now SHA1s of files in &lt;tt&gt;mongod-1::local&lt;/tt&gt;.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-1:/mongo/rs0_8/local# find . -type f -exec sha1sum {} \;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;a5ee9865a488720fefae8fb50d93dc59f02f5085  ./collection-0-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;aed1c55406f8c04f2d1ba66b1328fdd5c10dccf0  ./index-1-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;252b62bf3cd48434769cc77ca4a17255ec5b4b28  ./collection-2-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ac3c642b06a213a4897508d83087fbdf8e7bad71  ./index-3-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ac469e36e1f2a2d7b09aaa557939d6437a24321e  ./collection-4-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;f1d054442f0b15814000d0cbfce3336f43057e67  ./index-5-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;000adfebaa328c3e85a89627b6fa08b8b7eab88d  ./collection-6-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;41129285edc0117139d365c0b63313afa0a634d8  ./index-7-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;b467ba25df451e5a67404a9fcf3199e31d7b1f59  ./collection-8-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;8950463508ba036aff8b2657f5b1dc247a43d259  ./index-9-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;9b0e624ab42c8047836571c69d687191660925fc  ./collection-82-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;9175a4fd5af37e93d966b56a25a96087153239da  ./index-83-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;d2ee304f279051fb1f3d2700e4e58caa2339541a  ./collection-14-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3f4049692686ee877457aeb581d853ff16ad6162  ./index-15-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2067414" author="drarunl" created="Tue, 20 Nov 2018 04:11:53 +0000"  >&lt;p&gt;Hi Danny:&lt;/p&gt;

&lt;p&gt;I do appreciate your team and you to continue to look at the issue.&lt;/p&gt;

&lt;p&gt;Per your suggestion, I brought up the replica set again in a PSA configuration. This time too it crashed. But turns out I made a mistake along the way that may provide some clue.&lt;/p&gt;

&lt;p&gt;I have uploaded the log and relevant files. Here are their details:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;f15ad69720901051e8038d49f85a058cbf14fd9d  mongod-1-2018-11-20T03-20-59.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7b5778ef42a8e8f7afa691809ded69413c5f211e  mongod-2-2018-11-20T03-21-41.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;3f40af687662330147b8ea1294b600e3b37c7550  mongod-arbiter-0-2018-11-20T03-21-55.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Here is what we did.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Used the cloudmanager for everything. Everything was done on freshly booted machines.&#160; But they had all the old DB.&lt;/li&gt;
	&lt;li&gt;Machines: mongod-1 (PRIMARY), mongod-2 (SECONDARY), mongo-arbiter&lt;/li&gt;
	&lt;li&gt;Replica set name: RS1-2 (chose a different name from earlier; just in case).&lt;/li&gt;
	&lt;li&gt;Turns out in the initial launch, we had not mounted the DB drive on mongod-2.&lt;/li&gt;
	&lt;li&gt;The automation agent log shows attempts to launch and failing.&lt;/li&gt;
	&lt;li&gt;Then when I mounted the drive, the mongod process on mongod-2 started.&#160;&lt;/li&gt;
	&lt;li&gt;And it immediately crashed, and brought mongod-1 down with it.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Relevant IP addresses:&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;10.128.0.12 - mongod-1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;10.128.0.19 - mongod-2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;10.128.0.13 - mongod-arbiter&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;The error log says that the crash is due to &lt;tt&gt;Permission denied&lt;/tt&gt;&#160;on files on both servers. I checked the permissions, and they look good to me.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Here is log from &lt;tt&gt;mongod-1&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:09:51.000+0000 E STORAGE  [ftdc] WiredTiger error (13) [1542683391:916][4448:0x7f89d88b4700], WT_SESSION.open_cursor: __posix_fs_exist, 189: /mongo/rs&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;0_8//local/collection-16-8568352613365886055.wt: file-exist: stat: Permission denied Raw: [1542683391:916][4448:0x7f89d88b4700], WT_SESSION.open_cursor: __posix_fs_&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;exist, 189: /mongo/rs0_8//local/collection-16-8568352613365886055.wt: file-exist: stat: Permission denied&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:09:51.024+0000 E STORAGE  [thread179] WiredTiger error (13) [1542683391:24081][4448:0x7f89de8c0700], log-server: __directory_list_worker, 48: /mongo/r&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;s0_8//journal: directory-list: opendir: Permission denied Raw: [1542683391:24081][4448:0x7f89de8c0700], log-server: __directory_list_worker, 48: /mongo/rs0_8//journ&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;al: directory-list: opendir: Permission denied&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Here is log from &lt;tt&gt;mongod-2&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 E STORAGE [thread911] WiredTiger error (13) [1542683304:750754][4423:0x7f993a6a6700], log-server: __directory_list_worker, 48: /mongo/&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;rs0_6//journal: directory-list: opendir: Permission denied Raw: [1542683304:750754][4423:0x7f993a6a6700], log-server: __directory_list_worker, 48: /mongo/rs0_6//jou&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;rnal: directory-list: opendir: Permission denied&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 E STORAGE [thread911] WiredTiger error (13) [1542683304:750857][4423:0x7f993a6a6700], log-server: __log_prealloc_once, 469: log pre-al&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;loc server error: Permission denied Raw: [1542683304:750857][4423:0x7f993a6a6700], log-server: __log_prealloc_once, 469: log pre-alloc server error: Permission deni&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ed&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 E STORAGE [thread911] WiredTiger error (13) [1542683304:750868][4423:0x7f993a6a6700], log-server: __log_server, 1015: log server error&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;: Permission denied Raw: [1542683304:750868][4423:0x7f993a6a6700], log-server: __log_server, 1015: log server error: Permission denied&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 E STORAGE [thread911] WiredTiger error (-31804) [1542683304:750877][4423:0x7f993a6a6700], log-server: __wt_panic, 523: the process mus&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;t exit and restart: WT_PANIC: WiredTiger library panic Raw: [1542683304:750877][4423:0x7f993a6a6700], log-server: __wt_panic, 523: the process must exit and restart&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;: WT_PANIC: WiredTiger library panic&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 F - [thread911] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 409&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-20T03:08:24.750+0000 F - [thread911]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The &lt;tt&gt;mongod-1:local&lt;/tt&gt;&#160;directory. The error above says the &lt;tt&gt;collection-16&lt;/tt&gt;&#160;&lt;tt&gt;file exists&lt;/tt&gt;, implying may be it shouldn&apos;t.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-1:~# ls -l /mongo/rs0_8/local/*&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        53248 Nov 19 22:37 /mongo/rs0_8/local/collection-0-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 20 02:33 /mongo/rs0_8/local/collection-14-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb 124586246144 Nov 19 22:36 /mongo/rs0_8/local/collection-16-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 Nov 19 22:36 /mongo/rs0_8/local/collection-2-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 20 03:08 /mongo/rs0_8/local/collection-4-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 19 22:36 /mongo/rs0_8/local/collection-6-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 19 22:36 /mongo/rs0_8/local/collection-80-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 19 22:36 /mongo/rs0_8/local/collection-82-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 19 22:36 /mongo/rs0_8/local/collection-8-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 May  1  2018 /mongo/rs0_8/local/index-15-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        36864 Nov 19 22:37 /mongo/rs0_8/local/index-1-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 May  1  2018 /mongo/rs0_8/local/index-3-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 May  1  2018 /mongo/rs0_8/local/index-5-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 Nov 19 22:36 /mongo/rs0_8/local/index-7-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 Nov 19 22:36 /mongo/rs0_8/local/index-81-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 May  1  2018 /mongo/rs0_8/local/index-83-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb mongodb        16384 May  1  2018 /mongo/rs0_8/local/index-9-8568352613365886055.wt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;And the &lt;tt&gt;mongod-1::journal&lt;/tt&gt;&#160;directory. The error says permission denied listing the directory. The processes were running as &lt;tt&gt;mongodb&lt;/tt&gt; user. A fact I further verified by the ownership of &lt;tt&gt;/tmp/mongod-6655.sock&lt;/tt&gt;. So I donot understand why permission would be denied listing the directory.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-1:~# ls -al /mongo/rs0_8//journal/&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;total 307220&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;drwx------  2 mongodb mongodb       110 Nov 19 22:36 .&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;drwxr-x--- 19 mongodb mongodb     16384 Nov 20 03:08 ..&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw-------  1 mongodb nogroup 104857600 Nov 20 03:09 WiredTigerLog.0000062081&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw-------  1 mongodb nogroup 104857600 Nov 19 22:36 WiredTigerPreplog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw-------  1 mongodb nogroup 104857600 Nov 19 22:36 WiredTigerPreplog.0000000002&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;On &lt;tt&gt;mongod-2&lt;/tt&gt; the error is only about permission denied listing the&#160;&lt;tt&gt;journal&lt;/tt&gt;&#160;directory. Here are permissions in that directory.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:~# ls -l /mongo/rs0_6/journal/&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;total 307200&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup 104857600 Nov 20 03:07 WiredTigerLog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup 104857600 Nov 20 03:00 WiredTigerPreplog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-rw------- 1 mongodb nogroup 104857600 Nov 20 03:00 WiredTigerPreplog.0000000002&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;On a whim I also computed the SHA1 of the two journal directories. Turns out the Preplog files on both servers are the same, though they differ in the time stamps. Their time stamps are indicators of about when each server was started.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-1:~#  find  /mongo/rs0_8/journal/ -type f -exec sha1sum {} \;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;31c9a529772abec60ef3121227b4db3455cf17fc  /mongo/rs0_8/journal/WiredTigerLog.0000062081&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4aebdfaa295761d1bce1f996be6051b3a6e9990f  /mongo/rs0_8/journal/WiredTigerPreplog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4aebdfaa295761d1bce1f996be6051b3a6e9990f  /mongo/rs0_8/journal/WiredTigerPreplog.0000000002&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:~# find  /mongo/rs0_6/journal/ -type f -exec sha1sum {} \;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;b86612183c99e587cd8a3b0e708c3ffa3486eccf  /mongo/rs0_6/journal/WiredTigerLog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4aebdfaa295761d1bce1f996be6051b3a6e9990f  /mongo/rs0_6/journal/WiredTigerPreplog.0000000001&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;4aebdfaa295761d1bce1f996be6051b3a6e9990f  /mongo/rs0_6/journal/WiredTigerPreplog.0000000002&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;root@mongod-2:~# &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="2066196" author="daniel.hatcher" created="Mon, 19 Nov 2018 14:35:49 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;We&apos;ve done some more investigation into the data you&apos;ve provided thus far. We are still looking into why the oldest timestamp did not advance once the node applied all of its oplog. However, we believe that some of the improvements in later versions of 4.0.x have improved memory usage for situations such as this so that you can get around the OOM. Would it be possible to try running the node in the replica set using the &lt;a href=&quot;https://www.mongodb.com/download-center/community&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;latest 4.0.4 binaries&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;Thank you very much,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2065334" author="daniel.hatcher" created="Fri, 16 Nov 2018 22:15:16 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;Yes, the setting forces read-only mode on the server for technical reasons.&lt;/p&gt;

&lt;p&gt;We are seeing some unexplained activity on the server after the oplog batches have been applied. We are investigating as to what could be causing it. As such, I am currently unable to state equivocally that the data on the node is complete.&lt;/p&gt;

&lt;p&gt;That being said, I believe that all the issues have dealt with recovering data and all the data currently on the node should be safe (not corrupted). Do you have some way of validating the data presence on the client-side? You may wish to run the &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/validate/index.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validate command&lt;/a&gt; on the standalone as it should touch all the data files.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2064663" author="drarunl" created="Fri, 16 Nov 2018 16:43:37 +0000"  >&lt;p&gt;Uploaded logs and diagnostics. Configuration &#8211; standalone with the&#160;&#160;&lt;tt&gt;&#8211;-setParameter recoverFromOplogAsStandAlone=1&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;7411370a7d5a6ae0d9134d5608dd4ad4a1f86050 mongod-0-2018-11-16T16-39-11.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2064573" author="drarunl" created="Fri, 16 Nov 2018 15:54:23 +0000"  >&lt;p&gt;Hi Danny:&lt;/p&gt;

&lt;p&gt;Re: Saving all the data.&#160;&lt;/p&gt;

&lt;p&gt;It&apos;s a matter of trade-off I need to make.&lt;/p&gt;

&lt;p&gt;a) I have over 2 years of data. The oplog replay is for about a month.&lt;/p&gt;

&lt;p&gt;b) For over a week now I have completely shutdown my entire system &#8211; which is analyzing a stream of malware I get on a daily basis. Prior to that the system was flaky so I figured I should solve the core problem. As it is I do not know the state of the data. (My database schema has not been designed to account for intermittent failures.)&lt;/p&gt;

&lt;p&gt;Which means if I have to lose a month&apos;s worth of data to get back to a reliable operation today, as opposed to wait indefinite amount of time. I will choose losing the oplog data.&lt;/p&gt;

&lt;p&gt;However, at this time we have not ruled out the possibility that the data storage itself is corrupted. I have shutdown any query (more so insertions) in the data.&#160;&lt;/p&gt;

&lt;p&gt;So I am anxious to know whether the primary data storage has any issues.&lt;/p&gt;

&lt;p&gt;------&lt;/p&gt;

&lt;p&gt;Re: Status from using the &lt;tt&gt;--recoverFromOplogAsStandalone&lt;/tt&gt;&#160;switch.&lt;/p&gt;

&lt;p&gt;First, for anyone else following this thread, depending on your mongod version, you may need to use&#160; &lt;tt&gt;&#8211;-setParameter recoverFromOplogAsStandAlone=1&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;So far the server has been up and has not crashed, under no activity of any kind.&lt;/p&gt;

&lt;p&gt;------&lt;/p&gt;

&lt;p&gt;Observation:&lt;/p&gt;

&lt;p&gt;The banner message splashed by the server after restart with the switch says its running in &lt;tt&gt;readOnly&lt;/tt&gt;&#160;mode. I take that to mean that the DB will not permit any inserts. If so, this mode is good for me to get the data out, but not much else.&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;018-11-16T09:05:24.293+0000 W REPL [initandlisten] Setting mongod to readOnly mode as a result of specifying &apos;recoverFromOplogAsStandalone&apos;.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.370+0000 I STORAGE [initandlisten] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** Database contents may appear inconsistent with the oplog and may appear to not contain&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** writes that were visible when this node was running as part of a replica set.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** Restart with --replSet unless you are doing maintenance and no other clients are connected.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** The TTL collection monitor will not start because of this.&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] ** &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] For more info see http://dochub.mongodb.org/core/ttlcollections&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.372+0000 I STORAGE [initandlisten] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-11-16T09:05:24.414+0000 I NETWORK [initandlisten] waiting for connections on port 6666&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;MY PLAN:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;I will shutdown the server and post the logs.&#160; Not sure if that would create new&#160;&lt;tt&gt;diagnostics.data&lt;/tt&gt;. I will upload the most recent anyway.&lt;/li&gt;
	&lt;li&gt;I will run the server standalone, and open it up to normal processing for a few hours. In the process I will be exposing myself to some loss of data. Either the new data that I am creating or the old data in the oplog. But I will get piece of mind if I can rule out major corruption of data.&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2064453" author="daniel.hatcher" created="Fri, 16 Nov 2018 15:00:38 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;That flag is actually relatively new with MongoDB 4.0 as in previous versions we recovered from the journal; in 4.0 we journal the oplog and recover from the oplog after the journal.&lt;/p&gt;

&lt;p&gt;As you noticed, starting the node as a standalone did not replay the oplog. Thus, the data there could be data that has not been applied yet. We are operating under the assumption that those updates are valuable; if instead there has not been any real production data since the disk resizing then it should be easy to recover from this state but losing some of those writes. However, we will continue assuming that you wish to save all of the data.&lt;/p&gt;

&lt;p&gt;Starting the standalone with the &lt;tt&gt;--recoverFromOplogAsStandalone&lt;/tt&gt; flag may cause you to run into the same OOM issue again. But we do think it&apos;s the next step. Regardless of the outcome, please upload fresh &lt;tt&gt;diagnostic.data&lt;/tt&gt; to this ticket after the node has gone through the replay process.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2064134" author="drarunl" created="Fri, 16 Nov 2018 05:32:47 +0000"  >&lt;p&gt;Dennis:&lt;/p&gt;

&lt;p&gt;Thanks for the note of caution. I do have a backup of my two replicas. So I am good.&lt;/p&gt;

&lt;p&gt;I did run the server in standalone, it restarted in a jiffy. No oplog replay. And stayed up for long enough. But I didn&apos;t put it under any load. No queries, nothing.&lt;/p&gt;

&lt;p&gt;Follow up to your note I also go this hint from:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bruce.lucas&quot; class=&quot;user-hover&quot; rel=&quot;bruce.lucas&quot;&gt;bruce.lucas&lt;/a&gt;, the standalone node will replay the oplog if started with &lt;tt&gt;--recoverFromOplogAsStandalone&lt;/tt&gt;.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;This escaped my mind when I restarted the server. (Been a busy day with meetings, and I got both the notes when I was in a meeting.)&lt;/p&gt;

&lt;p&gt;So FWIW, I will do the standalone with the flag Bruce mentions.&#160; As and when you have something new for me to try, I can bring my other two servers back up.&lt;/p&gt;

&lt;p&gt;Arun&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2063688" author="daniel.hatcher" created="Thu, 15 Nov 2018 20:14:07 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;To keep you updated, we&#8217;ve been having an internal discussion on the best way forward and a few snippets have been getting through. Tess&#8217;s comment and Bruce&#8217;s graph were two of those. Unfortunately, the graph was generated by a tool that is not yet customer-available. There is a project in the works to make it so but it&apos;s still down-the-line.&lt;/p&gt;

&lt;p&gt;It may be best to keep Automation disabled for the entire replica set until we get your cluster back into a healthy state. In its attempts to fix the problem it is possible that Automation may contradict some of your actions.&lt;/p&gt;

&lt;p&gt;We are discussing the standalone idea as we are not sure if the content will still be pinned in cache if the node is no longer in the replica set. If you can hold off for a little, we&apos;d like to make sure that going to a standalone in the best option before recommending it. However, if you need to get your system up as soon as possible, please make sure that you have the data files backed up to a separate location just in case.&lt;/p&gt;

&lt;p&gt;We will let you know more as soon as we can. I appreciate your patience in working with us through this; getting to the root cause here will also help us support the rest of our customers.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2063346" author="drarunl" created="Thu, 15 Nov 2018 16:35:56 +0000"  >&lt;p&gt;Bruce: That is one cool graph, showing timeline. Is that an in-house tool, or something available to users.&lt;/p&gt;

&lt;p&gt;All:&#160; I ran the replica again with just PRIMARY and ARBITER. No SECONDARY. And I also turned off any other communication with the server. So absolutely no queries.&lt;/p&gt;

&lt;p&gt;After 3 hours of replaying oplog, and about 9 hours of staying up, the server crashed.&lt;/p&gt;

&lt;p&gt;CAVEATS: I did find that my automation manager was not able to change the replica config. Reason: the _id of the PRIMARY (mongod-0) in the replica was different from the _id being given by automation manager.&#160;&#160;&lt;/p&gt;

&lt;p&gt;So I turned off automation, and manually removed mongod-1 from the replica. Good thing is that mongod-1 was powered off all through, so mongod-0 was really by itself. Except, it was expecting mongod-1 in the replica. And so you see heartbeats to mongod-1 that fail.&lt;/p&gt;

&lt;p&gt;I have uploaded the log and diagnostics of both mongod-0 and mongod-arbiter. Here are the SHA1s of the pushed files.&lt;br/&gt;
 &#160;&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;6c09c42dacb8c0860b850dfe80102ef37a4058b8 mongod-0-2018-11-15T16-18-15.tgz &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;5a603937fe5c7a518100d4346c9fb5a7b07883ac mongod-arbiter-0-2018-11-15T16-21-47.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;&#160;&lt;br/&gt;
 &#160;&lt;br/&gt;
 In the diagnostic I have kept metrics.interim and the most recent metrics.$TIMESTAMP file. In the logs, I have kept the recent logs.&lt;br/&gt;
 &#160;&lt;br/&gt;
 I have a crude script to collect the necessary data and push it to the server. If it&apos;s of any value, I&apos;d be happy to share.&lt;br/&gt;
 &#160;&lt;br/&gt;
 MY PLAN:&lt;br/&gt;
 &#160;&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Run the server standalone. Do it manually (no cloud manager). Run with --repair.&lt;/li&gt;
	&lt;li&gt;Disconnect any access to the server (no queries)&lt;/li&gt;
	&lt;li&gt;Wait 9 hours.&#160;&lt;br/&gt;
 &#160;&lt;br/&gt;
 &#160;&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="2062516" author="drarunl" created="Wed, 14 Nov 2018 23:21:55 +0000"  >&lt;p&gt;I received a note from Tess Avitable. Don&apos;t see it in the ticket here. For reference, here is its:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;When read concern majority is disabled, we don&apos;t update the stable timestamp, but we do update the oldest timestamp, which controls how much history is pinned in memory.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Good observation. I had disabled &quot;enableMajorityReadConcern&quot; while shooting in the dark, and trying to do something. I found a reference to the property on stack exchange, and then found this description on the docs. I thought of giving it a chance.&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;you can disable read concern&#160;&lt;a href=&quot;https://docs.mongodb.com/manual/reference/read-concern-majority/#readconcern.%22majority%22&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;&quot;majority&quot;&lt;/tt&gt;&lt;/a&gt;&#160;to prevent the storage cache pressure from immobilizing a deployment with a three-member primary-secondary-arbiter (PSA) architecture&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;I figured by deployment was immobilized, so give it a try.&#160;&lt;/p&gt;

&lt;p&gt;I have now enabled it back (by deleting the property on cloudmanager, so it reverts to the default).&#160;&lt;/p&gt;

&lt;p&gt;My plan is to still bring up the PRIMARY up first. Let it run for sometime, and then bring up the SECONDARY.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2061768" author="drarunl" created="Wed, 14 Nov 2018 15:34:45 +0000"  >&lt;p&gt;Kelsey:&lt;/p&gt;

&lt;p&gt;Thanks for the upload location.&lt;/p&gt;

&lt;p&gt;I have uploaded the files from my CURRENT configuration. However, there may be a major issue in using this data. Yesterday we &quot;disk copied&quot; PRIMARY (mongod-0)&#160; to create the SECONDARY (mongod-1). Which means a good bit of the history on mongod-1 is from mongod-0.&lt;/p&gt;

&lt;p&gt;FWIW, I have also included mongod.log files from around October 22 as well. That is when we resized the disk, and the problems started. In case it is useful, the replay oplog starts apply oplogs from October 17.&lt;/p&gt;

&lt;p&gt;I do have a copy of the old mongod-1. I&apos;d need to get the data out from it for you.&lt;/p&gt;

&lt;p&gt;Here are the sha1sums of the files I uploaded today.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;9a18047b359ca3b332e0450c208acbb802f6e1e8 mongod-1.tgz&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;bdaf9bb331288ac5ccf046d99ee64fa9a3bc42d1 mongod-0.tgz&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;e8de78c5e0053b4992f06a5f98a7b1ddd13e2334 mongod-arbiter-0.tgz&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;EDIT: Uploaded&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;5778a9a467a89314c1cb0e9d596d43fe4bca060e mongod-2.tgz&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;RECAP: My early configuration where this problem started had mongod-0 as PRIMARY and mongod-2 as SECONDARY (but at that time it was called mongod-1).&lt;/p&gt;

&lt;p&gt;The configuration setup yesterday had mongod-0 as PRIMARY and mongod-1 as SECONDARY (where mongod-1 was created by disk copy from mongod-0).&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Arun&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2061247" author="thomas.schubert" created="Wed, 14 Nov 2018 10:00:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=DrArunL&quot; class=&quot;user-hover&quot; rel=&quot;DrArunL&quot;&gt;DrArunL&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Would you please upload a more recent archive of diagnostic.data directory as well as the complete logs for the past three days for all the nodes in the replica set (including the arbiter)? I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/96a9dbf2-ad33-4257-bd87-41cd7c8b49ef.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure upload portal&lt;/a&gt; for you to use to provide these files to help you get around the size limitations of attachments to JIRA.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="2061124" author="drarunl" created="Wed, 14 Nov 2018 03:38:20 +0000"  >&lt;p&gt;Excitement short lived. The SECONDARY&#160; (mongod-1) created from copying PRIMARY (mongod-0) crashed due to memory allocation error.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;2018-11-14T02:50:44.488+0000 E STORAGE &lt;span class=&quot;error&quot;&gt;&amp;#91;WTCheckpointThread&amp;#93;&lt;/span&gt; WiredTiger error (12) &lt;span class=&quot;error&quot;&gt;&amp;#91;1542163844:402387&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;22365:0x7f2e68ca8700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file://wiredtigerlas.wt/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTigerLAS.wt&lt;/a&gt;, WT_CURSOR.next: memory allocation of 27665 bytes failed: Cannot allocate memory Raw: &lt;span class=&quot;error&quot;&gt;&amp;#91;1542163844:402387&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;22365:0x7f2e68ca8700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file://wiredtigerlas.wt/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTigerLAS.wt&lt;/a&gt;, WT_CURSOR.next: memory allocation of 27665 bytes failed: Cannot allocate memory&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;This then triggers cascading errors, and the server restarts.&lt;/p&gt;

&lt;p&gt;I have attached the recent log. Extracted only the lines between two SERVER RESTARTED lines.&lt;/p&gt;

&lt;p&gt;When the server restarts, I see that it goes through the &quot;replaying stored operations&quot;, which takes almost 3 hours. As I write, mongod-1 is going through that sequence again. Am curious if there is a way for making the server remember the stored operatons and not replay it again.&lt;/p&gt;

&lt;p&gt;EDIT: Turns out both the PRIMARY (mongod-0) and SECONDARY (mongod-1) crashed at exactly the same time (within a minute of each other). I have attached log of both.&lt;/p&gt;

&lt;p&gt;The two log files show a lot of network connections. The following may be helpful to understand the connections.&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;10.128.0.10 - mongod-0&#160;mongod-0.c.virusbattle-1.internal&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;10.128.0.12 - mongod-1&#160;mongod-1.c.virusbattle-1.internal&lt;/tt&gt;&lt;br/&gt;
 &lt;tt&gt;10.128.0.13 - arbiter&#160;mongod-arbiter-0.c.virusbattle-1.internal&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;My plan:&lt;/p&gt;

&lt;p&gt;Turn off the secondary. Let the primary run by itself for a day, maybe two. See if it croaks.&#160; If it does or doesn&apos;t should help isolate the issue further.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2060946" author="drarunl" created="Wed, 14 Nov 2018 00:15:12 +0000"  >&lt;p&gt;Went through the steps. I now have the PSA configuration back up, in that the mongod processes are running, and accepting connections (as indicated in the mongod.log files).&lt;/p&gt;

&lt;p&gt;However, the cloud manager shows there is problem. It says &quot;Wait until this process is primary&quot; (for mongod-0). I looked in the mongod.log file, and it shows that the process has indieed transitioned to PRIMARY. And mongod-1 also says that mongod-0 is PRIMARY.&lt;/p&gt;

&lt;p&gt;There is however an E (ERROR) message in the mongod-0 log file and a recurring I (INFO) in the mongod-1 log file which may hold some clue.&#160; The mongod-0 error is also referenced in the &quot;View Details&quot; error log of the automation agent.&lt;/p&gt;

&lt;p&gt;Here is one instance of the error on mongod-0 (PRIMARY)&lt;/p&gt;

&lt;p&gt;{{2018-11-13T23:45:20.660+0000 E REPL &lt;span class=&quot;error&quot;&gt;&amp;#91;conn5585&amp;#93;&lt;/span&gt; replSetReconfig got NewReplicaSetConfigurationIncompatible: New and old configurations both have members with host of mongod-0.c.virusbattle-1.internal:6655 but in the new configuration the _id field is 0 and in the old configuration it is 8 for replica set rs0 while validating { _id: &quot;rs0&quot;, version: 30, members: [ { priority: 2.0, votes: 1, arbiterOnly: false, buildIndexes: true, _id: 0, host: &quot;mongod-0.c.virusbattle-1.internal:6655&quot;, hidden: false, tags: {}, slaveDelay: 0 }, { hidden: false, host: &quot;mongod-arbiter-0.c.virusbattle-1.internal:6655&quot;, tags: {}, _id: 2, priority: 0.0, buildIndexes: true, arbiterOnly: true, votes: 1, slaveDelay: 0 }, { _id: 3, tags: {}, arbiterOnly: false, hidden: false, host: &quot;mongod-1.c.virusbattle-1.internal:6655&quot;, priority: 1.0, votes: 1, buildIndexes: true, slaveDelay: 0 } ], settings: { replicaSetId: ObjectId(&apos;5a4c05bb0f58fd8f4ca635e0&apos;), getLastErrorDefaults: &lt;/p&gt;
{ w: 1, wtimeout: 0 }
&lt;p&gt;, getLastErrorModes: {}, electionTimeoutMillis: 10000.0, chainingAllowed: true, catchUpTimeoutMillis: -1.0, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10.0, catchUpTakeoverDelayMillis: 30000.0 }, protocolVersion: 1 }}}&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;And the INFO entry in mongod-1 (SECONDARY) &#8211; For reference, this server was restarted on 20:16. And these are some of the messages.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;2018-11-13T23:36:03.194+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;LogicalSessionCacheRefresh&amp;#93;&lt;/span&gt; Starting new replica set monitor for rs0/mongod-0.c.virusbattle-1.internal:6655,mongod-1.c.virusbattle-1.internal:6655&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-13T23:36:03.196+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;LogicalSessionCacheRefresh&amp;#93;&lt;/span&gt; Successfully connected to mongod-0.c.virusbattle-1.internal:6655 (11 connections now open to mongod-0.c.virusbattle-1.internal:6655 with a 0 second timeout)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-13T23:36:03.196+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;LogicalSessionCacheReap&amp;#93;&lt;/span&gt; Successfully connected to mongod-0.c.virusbattle-1.internal:6655 (12 connections now open to mongod-0.c.virusbattle-1.internal:6655 with a 0 second timeout)&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-13T23:36:03.196+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;LogicalSessionCacheRefresh&amp;#93;&lt;/span&gt; scoped connection to mongod-0.c.virusbattle-1.internal:6655 not being returned to the pool&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;2018-11-13T23:36:03.196+0000 I NETWORK &lt;span class=&quot;error&quot;&gt;&amp;#91;LogicalSessionCacheReap&amp;#93;&lt;/span&gt; scoped connection to mongod-0.c.virusbattle-1.internal:6655 not being returned to the pool&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2059932" author="daniel.hatcher" created="Tue, 13 Nov 2018 13:45:40 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;That&apos;s actually our recommended strategy if you think that the MongoDB&apos;s initial sync process is too slow. It&apos;s mentioned in our documentation on &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/index.html#replica-set-resync-by-copying&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;resyncing a member of a replica set&lt;/a&gt;. The procedure is:&lt;/p&gt;

&lt;p&gt;1. Stop the Secondary process&lt;br/&gt;
2. Remove all the files under the Secondary&apos;s &lt;tt&gt;dbpath&lt;/tt&gt;&lt;br/&gt;
3. Stop the Primary process&lt;br/&gt;
4. Copy the data files from the Primary&apos;s &lt;tt&gt;dbpath&lt;/tt&gt; to the Secondary&apos;s &lt;tt&gt;dbpath&lt;/tt&gt;&lt;br/&gt;
5. Start the Primary&lt;br/&gt;
6. Start the Secondary&lt;/p&gt;

&lt;p&gt;Once the Secondary is started again, it should apply everything in the oplog that it didn&apos;t have from the data copy (inserts/updates between steps 5 and 6). Then it will be a healthy member again.&lt;/p&gt;

&lt;p&gt;Please note that because you are using Cloud Manager Automation you must first remove these processes from automation before performing the above procedure. Otherwise, as you mentioned in your initial description, Automation will automatically try to start the processes if it notices that they are down.&lt;/p&gt;

&lt;p&gt;Please let me know if you have any issues with this.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2059651" author="drarunl" created="Tue, 13 Nov 2018 03:48:42 +0000"  >&lt;p&gt;Hi Danny:&lt;/p&gt;

&lt;p&gt;Thanks for the suggestion about initial sync. That seems like a reset button for mongod. Am newbie in this.&#160; Will try it in the morning tomorrow.&lt;/p&gt;

&lt;p&gt;In regards making a replica, what if I stopped my PRIMARY, copied the drive,&#160; and used that to start by SECONDARY.&lt;/p&gt;

&lt;p&gt;I am guessing that&apos;s a no-no. But I thought I&apos;d ask. I think copying disk may be faster than a init sync.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="2059328" author="daniel.hatcher" created="Mon, 12 Nov 2018 20:56:36 +0000"  >&lt;p&gt;Hello Arun,&lt;/p&gt;

&lt;p&gt;Your summary makes sense to me. Have you attempted to perform an initial sync since the problem on 4.0.1? If not, could you do so on this node that is currently experiencing the fatal assertion? It&apos;s possible that the oplog for it is in an unrecoverable state.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="585014">SERVER-36495</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="200958" name="image-2018-11-13-21-54-57-681.png" size="344739" author="DrArunL" created="Wed, 14 Nov 2018 03:54:57 +0000"/>
                            <attachment id="200715" name="mongod-0.tgz" size="82780558" author="DrArunL" created="Mon, 12 Nov 2018 15:58:25 +0000"/>
                            <attachment id="200714" name="mongod-1.tgz" size="62003583" author="DrArunL" created="Mon, 12 Nov 2018 15:58:23 +0000"/>
                            <attachment id="200957" name="mongodb-0.log-2018-11-14" size="31579822" author="DrArunL" created="Wed, 14 Nov 2018 03:51:44 +0000"/>
                            <attachment id="200956" name="mongodb-1.log-2018-11-14" size="6844631" author="DrArunL" created="Wed, 14 Nov 2018 03:31:22 +0000"/>
                            <attachment id="201090" name="node1.png" size="320112" author="bruce.lucas@mongodb.com" created="Thu, 15 Nov 2018 15:40:22 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>31.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 12 Nov 2018 20:56:36 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 9 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1310</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 9 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>DrArunL</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hucnfb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hu2unj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;I have the various log, turtle, wt, and diagnostic data files. But I have not idea how to recreate it.&#160;&lt;/p&gt;

&lt;p&gt;Am attaching all the pertinent data,&#160; I could think of.&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;mongod-1.tgz: Secondary (with the problem)&lt;/li&gt;
	&lt;li&gt;mongod-0.tgz: Primary&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;We are using monogdb cloud manager. The tar file contains logs of the automation agents as well.&lt;/p&gt;

&lt;p&gt;Since the automation agent keeps trying to restart the server, you&apos;d find repetitive entries. It contains errors from 4.0.1 and from 4.0.4.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huc9on:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>