<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:05:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-66467] WiredTiger checksum error</title>
                <link>https://jira.mongodb.org/browse/SERVER-66467</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;After repairing the db due to disk corruption, the db kept failing due to this error:&lt;/p&gt;

&lt;p&gt;2022-05-14T10:19:35.986+0000 E &#160;STORAGE &#160;&lt;span class=&quot;error&quot;&gt;&amp;#91;conn1137&amp;#93;&lt;/span&gt; WiredTiger error (0) &lt;span class=&quot;error&quot;&gt;&amp;#91;1652523575:986492&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;1:0x7f62e8b4d700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:///collection-1792-6427612125299872100.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-1792-6427612125299872100.wt&lt;/a&gt;, WT_CURSOR.next: __wt_block_read_off, 283: collection-1792-6427612125299872100.wt: read checksum error for 12288B block at offset 14536716288: block header checksum of 0xf0e3571e doesn&apos;t match expected checksum of 0xd8f2e206 Raw: &lt;span class=&quot;error&quot;&gt;&amp;#91;1652523575:986492&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;1:0x7f62e8b4d700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:///collection-1792-6427612125299872100.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-1792-6427612125299872100.wt&lt;/a&gt;, WT_CURSOR.next: __wt_block_read_off, 283: collection-1792-6427612125299872100.wt: read checksum error for 12288B block at offset 14536716288: block header checksum of 0xf0e3571e doesn&apos;t match expected checksum of 0xd8f2e206&lt;br/&gt;
2022-05-14T10:19:35.986+0000 E &#160;STORAGE &#160;&lt;span class=&quot;error&quot;&gt;&amp;#91;conn1137&amp;#93;&lt;/span&gt; WiredTiger error (0) &lt;span class=&quot;error&quot;&gt;&amp;#91;1652523575:986814&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;1:0x7f62e8b4d700&amp;#93;&lt;/span&gt;, &lt;a href=&quot;file:///collection-1792-6427612125299872100.wt&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;file:collection-1792-6427612125299872100.wt&lt;/a&gt;, WT_CURSOR.next: __wt_bm_corrupt_dump, 135: {14536716288, 12288, 0xd8f2e206}: (chunk 1 of 12): 00 00 00 00 00 00 00 00 71 5a 80 00 00 00 00 00 b7 1a 01 00 06 00 00 00 07 05 00 00 00 40 00 00 1e 57 e3 f0 01 00 00 00 11 e3 3f c9 17 80 e2 1f 05 85 3f 00 00 07 5f 69 64 00 61 c0 dc dd f7 b7 23 3d 00 00 00 00 00 00 f7 b4 04 f0 3c 08 1f 5c 7f 83 65 02 75 73 65 72 6e 61 6d 65 00 0f 00 00 00 32 39 30 31 31 30 37 31 35 30 30 30 39 3.........&lt;/p&gt;</description>
                <environment></environment>
        <key id="2046705">SERVER-66467</key>
            <summary>WiredTiger checksum error</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="chris.kelly@mongodb.com">Chris Kelly</assignee>
                                    <reporter username="ahmed.nasr@fixedmea.com">Ahmed Nasr</reporter>
                        <labels>
                    </labels>
                <created>Sat, 14 May 2022 14:04:50 +0000</created>
                <updated>Thu, 2 Jun 2022 14:06:02 +0000</updated>
                            <resolved>Wed, 18 May 2022 22:55:39 +0000</resolved>
                                    <version>4.2.5</version>
                                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4558719" author="JIRAUSER1269347" created="Thu, 19 May 2022 09:25:06 +0000"  >&lt;p&gt;Hi Chris,&lt;br/&gt;
Thanks for your help and insights. I wish you the best.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Ahmed&lt;/p&gt;</comment>
                            <comment id="4557591" author="JIRAUSER1265262" created="Wed, 18 May 2022 19:44:44 +0000"  >&lt;p&gt;Hi Ahmed,&lt;/p&gt;

&lt;p&gt;Thanks for providing a good timeline of events on this. Without logs, we can&apos;t really discern much more information, but there is a chance that the primary node is sustaining some sort of persistent issue that is happening irregularly. This could&apos;ve led to latent corruption on the node that just happened to be a problem when the secondary required an initial sync, requiring the primary to visit pages with corrupt data (and then causing the error you mentioned).&lt;/p&gt;

&lt;p&gt;Among corruption cases, checksum failures are usually caused by file system or a disk issues, so we are somewhat suspect of glusterfs. &lt;a href=&quot;https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/3.0_update_3_release_notes/chap-known_issues&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;There appear to be ways this can lead to data loss in certain situations.&lt;/a&gt; We can&apos;t speak much to it, but there appears to be a few examples of corruption that happen to those using Openshift with glusterfs in the past, including a mention of it taking place in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-40088&quot; title=&quot;WiredTiger has failed to open its metadata&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-40088&quot;&gt;&lt;del&gt;SERVER-40088&lt;/del&gt;&lt;/a&gt; with other database systems.&lt;/p&gt;

&lt;p&gt;Since we don&apos;t have much more information to go off of, and you have already resolved the issue, I&apos;m going to go ahead and close this ticket for now.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Christopher&lt;/p&gt;</comment>
                            <comment id="4554518" author="JIRAUSER1269347" created="Tue, 17 May 2022 20:17:46 +0000"  >&lt;p&gt;Hi Chris,&lt;/p&gt;

&lt;p&gt;Our cluster consists of 3 nodes, 1 primary, 1secondary and 1 arbiter.&lt;/p&gt;

&lt;p&gt;The cluster is deployed on openshift with glusterfs as the underlying file system.&lt;/p&gt;

&lt;p&gt;What happened is:&lt;/p&gt;

&lt;p&gt;1- The secondary node failed due to similar error.&lt;/p&gt;

&lt;p&gt;2- The whole volume of the secondary was removed and the node was started again with a clean volume.&lt;/p&gt;

&lt;p&gt;3- The secondary joined the cluster and started syncing from the primary.&lt;/p&gt;

&lt;p&gt;4- During syncing the primary failed with the corruption error.&lt;/p&gt;

&lt;p&gt;5- The secondary then failed as it didn&apos;t sync enough data to be able to act as primary.&lt;/p&gt;

&lt;p&gt;6- The primary was restarted automatically and was healthy.&lt;/p&gt;

&lt;p&gt;7- Then the cycle from 3 to 6 kept repeating.&lt;/p&gt;</comment>
                            <comment id="4554317" author="JIRAUSER1265262" created="Tue, 17 May 2022 19:24:35 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahmed.nasr%40fixedmea.com&quot; class=&quot;user-hover&quot; rel=&quot;ahmed.nasr@fixedmea.com&quot;&gt;ahmed.nasr@fixedmea.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It&apos;s hard to say without logs which would include exact information about your setup. Corruption can happen in any number of ways. &lt;b&gt;If you still have logs of this event please upload them to the support link if you can.&lt;/b&gt;&#160;&lt;/p&gt;

&lt;p&gt;We would be especially interested in figuring out why your node&apos;s initial sync failed. Did you get the same exact error you reported mid-sync on the fresh node? Or did it happen on the other node at any point? If you could provide a clearer timeline of the events that would be helpful for future issues.&lt;/p&gt;

&lt;p&gt;However, we do have some guidelines that should cover some common reasons for this.&lt;/p&gt;

&lt;p&gt;To avoid a problem like this in the future, it is our strong recommendation to:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Use a &lt;a href=&quot;https://docs.mongodb.com/manual/replication/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;replica set&lt;/a&gt;. (Which you are it sounds like)&lt;/li&gt;
	&lt;li&gt;Use the &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;most recent version&lt;/a&gt;. (I would recommend switching to a newer version; 4.2 is nearing EOL in April 2023)&lt;/li&gt;
	&lt;li&gt;Keep up to date &lt;a href=&quot;https://docs.mongodb.com/manual/core/backups/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;backups of your databases&lt;/a&gt;. (Which you did - great)&lt;/li&gt;
	&lt;li&gt;Follow all &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;production notes&lt;/a&gt;, especially those for &lt;a href=&quot;https://docs.mongodb.com/manual/administration/production-notes/#disk-and-storage-systems&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;underlying storage systems&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;Schedule and perform regular checks of the integrity of your filesystems and disks.&lt;/li&gt;
	&lt;li&gt;Never manipulate the underlying database files in any way.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;Regards,&lt;br/&gt;
Christopher&lt;/p&gt;</comment>
                            <comment id="4554140" author="JIRAUSER1269347" created="Tue, 17 May 2022 18:33:02 +0000"  >&lt;p&gt;Hi Chris,&lt;/p&gt;

&lt;p&gt;Thanks for your support. Unfortunately it was a major incident and we couldn&apos;t afford to wait. We tried repairing but unfortunately it stated that it has to sync from another node but it was down. Actually it was still re syncing from this node and the node failed due to that error half way. We had to restore a previous backup on a clean new replica set.&lt;/p&gt;

&lt;p&gt;But for future reference, what do you think might have caused such corruption? So we would watch for it in our new deployment?&lt;/p&gt;</comment>
                            <comment id="4554095" author="JIRAUSER1265262" created="Tue, 17 May 2022 18:23:31 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ahmed.nasr%40fixedmea.com&quot; class=&quot;user-hover&quot; rel=&quot;ahmed.nasr@fixedmea.com&quot;&gt;ahmed.nasr@fixedmea.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;The ideal resolution is to perform a clean &lt;a href=&quot;https://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#automatically-sync-a-member&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;resync&lt;/a&gt; from an unaffected node. In your case, I&apos;d recommend that next if you are running a replica set.&lt;/p&gt;

&lt;p&gt;You can also try &lt;a href=&quot;https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-mongod-repair&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;tt&gt;mongod --repair&lt;/tt&gt;&lt;/a&gt; using the latest patch of your version (in your case, 4.2.20).&lt;/p&gt;

&lt;p&gt;In the event that running &lt;tt&gt;--repair&lt;/tt&gt; using 4.2.20 is unsuccessful, then please provide the following:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;The logs leading up to the first occurrence of any issue&lt;/li&gt;
	&lt;li&gt;The logs of the repair operation.&lt;/li&gt;
	&lt;li&gt;The logs of any attempt to start mongod after the repair operation completed.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Would you please archive (tar or zip) the mongod.log files and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNjMxNzc3MzIxMzYiLCJleHAiOjE2NTU0MDM4NDl9.mJW_JTbd3LXHPGpm5lmWM84xZsd5O4jngqUmAbWWU-0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location?&lt;/p&gt;

&lt;p&gt;Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;


&lt;p&gt;Regards,&lt;br/&gt;
Christopher&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="2046706">SERVER-66468</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="378930" name="WiredTiger.turtle" size="1236" author="JIRAUSER1268850" created="Sat, 14 May 2022 13:56:54 +0000"/>
                            <attachment id="378931" name="WiredTiger.wt" size="5767168" author="JIRAUSER1268850" created="Sat, 14 May 2022 13:58:58 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 May 2022 18:23:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 37 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 37 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>ahmed.nasr@fixedmea.com</customfieldvalue>
            <customfieldvalue>chris.kelly@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0uvdz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0ds34:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[chris.kelly@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i0uhjb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>