<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:26:43 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31340] Fatal Assertion 17441 at src\mongo\db\storage\mmap_v1\record_store_v1_base.cpp 282</title>
                <link>https://jira.mongodb.org/browse/SERVER-31340</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Repairing a corrupt MongoDB is failing due to a crash. I have a 400GB large MongoDB database consisting of 206 fraction files (mostly 1.99GB). File no 198 seems to be corrupt/broken. The the primary is using MMAPv1 as storage engine. To repair the database I tied the following:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Deleted the secondary replica set and replicated again via version 3.2.6 (production). This started creating WiredTyger file types as is desired. This lead to a *&lt;b&gt;crash&lt;/b&gt;* of MongoDB where the corruption was discovered in the first place.&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;Removed this large DB from Mongo to see if something else is broken. After removal of this large DB all other DBs replicate fine and are fully accessible. This indicates the installation and setup is still fine.&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;mongodump --host somehost:27017 --db large --out C:\db-mongo\large-backup\&lt;br/&gt;
    This lead to a crash of MongoDB version 3.4.7 (development machine).&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;mongod --repair --repairpath C:\db-mongo\large-repair\ --config &quot;D:\db-mongo\mongo-db-repair.conf&quot;&lt;br/&gt;
This lead to a crash while repairing &quot;large.198&quot; of MongoDB version 3.4.7 after running for 30 hours (development machine 4GHz i7 32GB RAM was unusable/frozen for anything else in that time).&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;mongo-db-repair.conf: &lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    dbpath          = F:\mongo-dbs&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    logpath         = d:\db-mongo\logs\mongo-rep1.log&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    port            = &lt;/span&gt;&lt;span style=&quot;color: #009900; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;27017&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    logappend       = &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    directoryperdb  = &lt;/span&gt;&lt;span style=&quot;color: #006699; font-weight: bold; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;true&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    storageEngine   = mmapv1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The last entry in the log (see attached together with related mini dump) before printing the crash info is:&lt;br/&gt;
&lt;tt&gt;Fatal Assertion 17441 at src\mongo\db\storage\mmap_v1\record_store_v1_base.cpp 282&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;The plan is to get the replication going again and then update production to 3.4. Due to MongoDB &lt;b&gt;crashing&lt;/b&gt; this is not possible at present.&lt;/p&gt;</description>
                <environment>CentOs 7</environment>
        <key id="423973">SERVER-31340</key>
            <summary>Fatal Assertion 17441 at src\mongo\db\storage\mmap_v1\record_store_v1_base.cpp 282</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="Daniel5">Daniel Putra [X]</reporter>
                        <labels>
                    </labels>
                <created>Wed, 6 Sep 2017 10:53:17 +0000</created>
                <updated>Wed, 8 Jul 2020 13:02:45 +0000</updated>
                            <resolved>Mon, 27 Nov 2017 20:35:58 +0000</resolved>
                                    <version>3.4.7</version>
                                                    <component>MMAPv1</component>
                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1685663" author="thomas.schubert" created="Fri, 29 Sep 2017 20:39:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Daniel5&quot; class=&quot;user-hover&quot; rel=&quot;Daniel5&quot;&gt;Daniel5&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Yes, I would recommend taking another file copy. This will give &lt;tt&gt;mongodump --repair&lt;/tt&gt; the best chance of success, however, please note that we cannot guarantee that this option will succeed, and, if it does suceed, some manual intervention may be required to removed duplicated documents.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1683373" author="daniel5" created="Wed, 27 Sep 2017 17:10:32 +0000"  >&lt;p&gt;Hi Kelsey&lt;/p&gt;

&lt;p&gt;Now the copy of the DB on my machine fails to start up. I guess it is related to the crash of the last repair process (see high up in this case) which crashed. I get the errors below in the log.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;2017-09-27T09:42:24.734-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;main&amp;#93;&lt;/span&gt; Trying to start Windows service &apos;MongoDB&apos;&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; MongoDB starting : pid=8580 port=27017 dbpath=F:\mongo-dbs 64-bit host=Daniel2016&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; targetMinOS: Windows 7/Windows Server 2008 R2&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; db version v3.4.7&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; OpenSSL version: OpenSSL 1.0.1u-fips  22 Sep 2016&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; allocator: tcmalloc&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; modules: none&lt;br/&gt;
2017-09-27T09:42:24.735-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; build environment:&lt;br/&gt;
2017-09-27T09:42:24.736-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;     distmod: 2008plus-ssl&lt;br/&gt;
2017-09-27T09:42:24.736-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;     distarch: x86_64&lt;br/&gt;
2017-09-27T09:42:24.736-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;     target_arch: x86_64&lt;br/&gt;
2017-09-27T09:42:24.736-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; options: { config: &quot;D:\db-mongo\mongo-dev.conf&quot;, net: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { port}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;, service: true, setParameter: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { cursorTimeoutMillis}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;, storage: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { dbPath}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt;, systemLog: &lt;/p&gt;
&lt;div class=&quot;error&quot;&gt;&lt;span class=&quot;error&quot;&gt;Unknown macro: { destination}&lt;/span&gt; &lt;/div&gt;
&lt;p&gt; }&lt;br/&gt;
2017-09-27T09:42:24.736-0700 W -        &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Detected unclean shutdown - F:\mongo-dbs\mongod.lock is not empty.&lt;br/&gt;
2017-09-27T09:42:24.741-0700 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; ************** &lt;br/&gt;
old lock file: F:\mongo-dbs\mongod.lock.  probably means unclean shutdown,&lt;br/&gt;
but there are no journal files to recover.&lt;br/&gt;
this is likely human error or filesystem corruption.&lt;br/&gt;
please make sure that your journal directory is mounted.&lt;br/&gt;
found 3 dbs.&lt;br/&gt;
see: &lt;a href=&quot;http://dochub.mongodb.org/core/repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://dochub.mongodb.org/core/repair&lt;/a&gt; for more information&lt;br/&gt;
*************&lt;br/&gt;
2017-09-27T09:42:24.742-0700 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; exception in initAndListen: 12596 old lock file, terminating&lt;br/&gt;
2017-09-27T09:42:24.742-0700 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: going to close listening sockets...&lt;br/&gt;
2017-09-27T09:42:24.742-0700 I NETWORK  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; shutdown: going to flush diaglog...&lt;br/&gt;
2017-09-27T09:42:24.742-0700 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;serviceStopWorker&amp;#93;&lt;/span&gt; now exiting&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;I now cannot run &lt;em&gt;mongodump --repair&lt;/em&gt; as the DB fails to start up.&lt;br/&gt;
I tried &lt;em&gt;mongod --repair&lt;/em&gt; but it froze my machine (spec further up) after 15 minutes.&lt;/p&gt;

&lt;p&gt;I can take another file copy of the DB from the server but this means shutting down production. As it takes quite some time to copy the 400GB I can schedule this for the weekend to minimise impact. Should I do that or is there another way?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Daniel&lt;/p&gt;</comment>
                            <comment id="1683147" author="thomas.schubert" created="Wed, 27 Sep 2017 14:54:10 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Daniel5&quot; class=&quot;user-hover&quot; rel=&quot;Daniel5&quot;&gt;Daniel5&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Please note that &lt;tt&gt;mongod --repair&lt;/tt&gt; and &lt;tt&gt;mongodump --repair&lt;/tt&gt; are different operations that utilize different repair algorithms. From the terminal you should see something like:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;$ mongodump --repair&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.266-0400	writing repair of admin.system.indexes to&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.266-0400		repair cursor found 1 document in admin.system.indexes&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.266-0400	done dumping admin.system.indexes (0 documents)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.266-0400	writing repair of admin.system.version to&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.267-0400		repair cursor found 1 document in admin.system.version&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.267-0400	done dumping admin.system.version (0 documents)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.267-0400	writing repair of test.foo to&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.267-0400		repair cursor found 1 document in test.foo&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-09-27T10:51:32.267-0400	done dumping test.foo (0 documents)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1682791" author="daniel5" created="Wed, 27 Sep 2017 05:29:12 +0000"  >&lt;p&gt;Hi Kelsey&lt;br/&gt;
Not sure what you mean by &lt;em&gt;output&lt;/em&gt;?&lt;br/&gt;
I have attached the log file as well as the mini dump. Please guide me what else you require.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Daniel&lt;/p&gt;</comment>
                            <comment id="1682223" author="thomas.schubert" created="Tue, 26 Sep 2017 17:22:50 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Daniel5&quot; class=&quot;user-hover&quot; rel=&quot;Daniel5&quot;&gt;Daniel5&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for answering Mark&apos;s questions. Unfortunately, in cases like this it is very difficult to identify the root cause of the corruption. Would you please provide the output of &lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongodump/#cmdoption-repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;mongodump --repair&lt;/tt&gt;&lt;/a&gt;?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1679349" author="daniel5" created="Fri, 22 Sep 2017 02:20:35 +0000"  >&lt;p&gt;Thank you Mark for you feedback and attention to the case.&lt;/p&gt;

&lt;p&gt;Unfortunately this is the only DB we have. I had to delete the other replica set member a while ago and then thought this is no problem as I can just re-sync it. This is now unfortunately not possible due to the problem described on this case. I need a solution to get replication going again even if it means there is a bit of data loss. At present we are also stuck on v3.2 as upgrading the DB is not possible either.&lt;/p&gt;

&lt;p&gt;&lt;ins&gt;Please find the answers to your questions below.&lt;/ins&gt;&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; Locally. Are the disks SSDs or HDDs?  &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; 2TB Seagate SATA HDD. What kind of RAID and/or volume management system are you using? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; No RAID, Linux Centos 7 ext4. The box is sometimes unresponsive in which case we switch it off completely and then reboot.&lt;/li&gt;
	&lt;li&gt;Would you please check the integrity of your disks? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; Done via badblocks, result is: Pass completed, 0 bad blocks found. (0/0/0 errors)&lt;/li&gt;
	&lt;li&gt;Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through. &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; It started as a MongoDB 2.4 DB for a few years. It was then upgraded to 3.2 in 2016.&lt;/li&gt;
	&lt;li&gt;Have you manipulated (copied or moved) the underlying database files? If so, was mongod running? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; Yes, the DB was originally on another server. Before copying mongod had been stopped. The data was copied onto an external USB3 hard drive and then shipped 1000 km via courier services. This was before the upgrade from v2.4.&lt;/li&gt;
	&lt;li&gt;Have you ever restored this instance from backups? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; No. Never made one besides copying the entire DB. We have changed this recently where we now execute mongodump. Unfortunately this is not possible for this DB as described on this case.&lt;/li&gt;
	&lt;li&gt;What method do you use to create backups? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; Stop mongod and then copy all files. This is done max once a year if not only every two years. We rely on the replica sets being the backup.&lt;/li&gt;
	&lt;li&gt;When was the underlying filesystem last checked and is it currently marked clean? &lt;font color=&quot;red&quot;&gt;&lt;b&gt;DP:&lt;/b&gt; &lt;/font&gt; Never checked the filesystem. Let me know if you need something else.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I am very much looking forward hearing from you.&lt;br/&gt;
Daniel&lt;/p&gt;</comment>
                            <comment id="1678845" author="mark.agarunov" created="Thu, 21 Sep 2017 16:29:48 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=Daniel5&quot; class=&quot;user-hover&quot; rel=&quot;Daniel5&quot;&gt;Daniel5&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report. Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to &lt;a href=&quot;https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;resync the affected node&lt;/a&gt; or &lt;a href=&quot;https://docs.mongodb.com/manual/core/backups/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;restore from a backup&lt;/a&gt; if possible.&lt;/p&gt;

&lt;p&gt;To get an understanding of how this may have happened, I&apos;d like to request some information:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using?&lt;/li&gt;
	&lt;li&gt;Would you please check the integrity of your disks?&lt;/li&gt;
	&lt;li&gt;Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through.&lt;/li&gt;
	&lt;li&gt;Have you manipulated (copied or moved) the underlying database files? If so, was mongod running?&lt;/li&gt;
	&lt;li&gt;Have you ever restored this instance from backups?&lt;/li&gt;
	&lt;li&gt;What method do you use to create backups?&lt;/li&gt;
	&lt;li&gt;When was the underlying filesystem last checked and is it currently marked clean?&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="415865">SERVER-30629</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="164765" name="2017-09-03 mongo-rep1 mongod repair crash.log" size="162129" author="Daniel5" created="Wed, 6 Sep 2017 10:51:13 +0000"/>
                            <attachment id="164764" name="mongod.2017-09-03T05-36-26.mdmp" size="341399" author="Daniel5" created="Wed, 6 Sep 2017 10:51:28 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5002K00000nqFdsQAE]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 21 Sep 2017 16:29:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 19 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 19 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>Daniel5</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.agarunov</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hte9n3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|ht64g7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htdvq7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>