<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:53:59 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-40088] WiredTiger has failed to open its metadata</title>
                <link>https://jira.mongodb.org/browse/SERVER-40088</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We run MongoDB 3.2 in an OpenShift cluster using the official RedHat pod (hence the not quite up-to-date version). For reasons unknown yet, the pod failed to stop cleanly, corrupting the WiredTiger data files.&lt;/p&gt;

&lt;p&gt;The error is basically the same as described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23346&quot; title=&quot;WiredTiger.wt File corrupted (yet another one)&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23346&quot;&gt;&lt;del&gt;SERVER-23346&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27777&quot; title=&quot;file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27777&quot;&gt;&lt;del&gt;SERVER-27777&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25770&quot; title=&quot;WiredTiger.wt File corrupted&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25770&quot;&gt;&lt;del&gt;SERVER-25770&lt;/del&gt;&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-28242&quot; title=&quot;WiredTiger.wt file corrupted after unclean shutdown, cannot recover&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-28242&quot;&gt;&lt;del&gt;SERVER-28242&lt;/del&gt;&lt;/a&gt; and possibly others. However, none of the tickets we found includes instructions on how to fix the problem ourselves.&lt;/p&gt;

&lt;p&gt;We have tried to run an image with 3.6, that did not help. When running a 4.0 image, the output is more and it tries to repair, but only to fail with &quot;** IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.6  before attempting an upgrade to 4.0&quot;.&lt;/p&gt;

&lt;p&gt;A file-based restore does not help either, mongodb fails to start with checksum errors. Our customer did not create dumps.&lt;/p&gt;

&lt;p&gt;Can you help us? The database is currently down...&lt;/p&gt;

&lt;p&gt;Reading from the other logs, I have attached some files in order to get repair files. Please indicate if those are enough.&lt;/p&gt;

&lt;p&gt;Best Regards&lt;/p&gt;

&lt;p&gt;Christian Cremer&lt;/p&gt;</description>
                <environment></environment>
        <key id="715365">SERVER-40088</key>
            <summary>WiredTiger has failed to open its metadata</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="daniel.hatcher@mongodb.com">Danny Hatcher</assignee>
                                    <reporter username="ccremer">Christian Cremer</reporter>
                        <labels>
                    </labels>
                <created>Tue, 12 Mar 2019 14:31:29 +0000</created>
                <updated>Tue, 12 Mar 2019 17:28:59 +0000</updated>
                            <resolved>Tue, 12 Mar 2019 17:28:59 +0000</resolved>
                                    <version>3.2.10</version>
                                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2178606" author="daniel.hatcher" created="Tue, 12 Mar 2019 17:28:52 +0000"  >&lt;p&gt;Chris,&lt;/p&gt;

&lt;p&gt;Thanks for letting us know the repair worked!&lt;/p&gt;

&lt;p&gt;Have a great day,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                            <comment id="2178465" author="ccremer" created="Tue, 12 Mar 2019 15:55:16 +0000"  >&lt;p&gt;Hello Daniel&lt;/p&gt;

&lt;p&gt;Thank you very much for the superfast response and repair files! The repair worked and the server is now up again. We&apos;re waiting for the customer to verify the data, but I don&apos;t expect any more troubles in this regard.&lt;/p&gt;

&lt;p&gt;Yes, that or running databases outside of Openshift altogether. We have had DB corruptions with PostgreSQL and other systems when used with GlusterFS in the past before. However, since we are running Openshift as a public cluster, we don&apos;t always know what our customers are running.&lt;/p&gt;

&lt;p&gt;Nevertheless, thanks again!&lt;/p&gt;

&lt;p&gt;Cheers, Chris&lt;/p&gt;</comment>
                            <comment id="2178384" author="daniel.hatcher" created="Tue, 12 Mar 2019 15:17:15 +0000"  >&lt;p&gt;Hello Christian,&lt;/p&gt;

&lt;p&gt;We recommend using replica sets for situations like this. If one node goes down due to corrupt data, you can perform a clean sync from a healthy node without a service interruption.&lt;/p&gt;

&lt;p&gt;I&apos;ve attached  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/211179/211179_repair-attempt.tar&quot; title=&quot;repair-attempt.tar attached to SERVER-40088&quot;&gt;repair-attempt.tar&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; to this ticket. Can you replace the files under the &lt;tt&gt;$dbpath&lt;/tt&gt; with these and let me know if the node is able to start up?&lt;/p&gt;

&lt;p&gt;Thank you,&lt;/p&gt;

&lt;p&gt;Danny&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="211166" name="WiredTiger.turtle" size="959" author="ccremer" created="Tue, 12 Mar 2019 14:29:22 +0000"/>
                            <attachment id="211165" name="WiredTiger.wt" size="126976" author="ccremer" created="Tue, 12 Mar 2019 14:29:22 +0000"/>
                            <attachment id="211164" name="WiredTigerLAS.wt" size="4096" author="ccremer" created="Tue, 12 Mar 2019 14:29:22 +0000"/>
                            <attachment id="211179" name="repair-attempt.tar" size="133120" author="daniel.hatcher@mongodb.com" created="Tue, 12 Mar 2019 15:16:46 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 12 Mar 2019 15:17:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        4 years, 48 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            4 years, 48 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>ccremer</customfieldvalue>
            <customfieldvalue>daniel.hatcher@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huqkav:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hug7if:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; MongoDB starting : pid=27 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=frame-4-bcfcl&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; db version v3.2.10&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; git version: 79d9b3ab5ce20f51c272b4411202710a082d0317&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; allocator: tcmalloc&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; modules: none&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; build environment:&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;     distarch: x86_64&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt;     target_arch: x86_64&lt;br/&gt;
2019-03-12T11:58:57.249+0000 I CONTROL  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; options: { config: &quot;/etc/mongod.conf&quot;, net: &lt;/p&gt;
{ bindIp: &quot;127.0.0.1&quot;, port: 27017 }
&lt;p&gt;, replication: { oplogSizeMB: 64 }, security: &lt;/p&gt;
{ authorization: &quot;disabled&quot; }
&lt;p&gt;, storage: { dbPath: &quot;/var/lib/mongodb/data&quot;, wiredTiger: { engineConfig: &lt;/p&gt;
{ cacheSizeGB: 1 }
&lt;p&gt; } }, systemLog: &lt;/p&gt;
{ quiet: true }
&lt;p&gt; }&lt;br/&gt;
2019-03-12T11:58:57.262+0000 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Detected data files in /var/lib/mongodb/data created by the &apos;wiredTiger&apos; storage engine, so setting the active storage engine to &apos;wiredTiger&apos;.&lt;br/&gt;
2019-03-12T11:58:57.271+0000 W -        &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Detected unclean shutdown - /var/lib/mongodb/data/mongod.lock is not empty.&lt;br/&gt;
2019-03-12T11:58:57.280+0000 W STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Recovering data from the last clean checkpoint.&lt;br/&gt;
2019-03-12T11:58:57.283+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),&lt;br/&gt;
2019-03-12T11:58:57.493+0000 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger (-31802) &lt;span class=&quot;error&quot;&gt;&amp;#91;1552391937:493089&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;27:0x7fc04b147e80&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:WiredTiger.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTiger.wt&lt;/a&gt;, connection: unable to read root page from &lt;a href=&quot;file:WiredTiger.wt:&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTiger.wt:&lt;/a&gt; WT_ERROR: non-specific WiredTiger error&lt;br/&gt;
2019-03-12T11:58:57.493+0000 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger (0) &lt;span class=&quot;error&quot;&gt;&amp;#91;1552391937:493208&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;27:0x7fc04b147e80&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:WiredTiger.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTiger.wt&lt;/a&gt;, connection: WiredTiger has failed to open its metadata&lt;br/&gt;
2019-03-12T11:58:57.493+0000 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger (0) &lt;span class=&quot;error&quot;&gt;&amp;#91;1552391937:493245&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;27:0x7fc04b147e80&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:WiredTiger.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTiger.wt&lt;/a&gt;, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk&lt;br/&gt;
2019-03-12T11:58:57.493+0000 E STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; WiredTiger (0) &lt;span class=&quot;error&quot;&gt;&amp;#91;1552391937:493262&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;27:0x7fc04b147e80&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:WiredTiger.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:WiredTiger.wt&lt;/a&gt;, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options&lt;br/&gt;
2019-03-12T11:58:57.513+0000 I -        &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error&lt;br/&gt;
2019-03-12T11:58:57.513+0000 I STORAGE  &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating&lt;br/&gt;
2019-03-12T11:58:57.514+0000 I CONTROL &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; dbexit: rc: 100&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|huq6k7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>