<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:17:05 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-28084] WT_PANIC: WiredTiger library panic during bulk update</title>
                <link>https://jira.mongodb.org/browse/SERVER-28084</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;When doing a lot of of update operations, mongod processes randomly crash (not always, not often).&lt;/p&gt;

&lt;p&gt;This is a development shard server (with core sharding) :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;3 shards of 3 members each (Primary, Secondary, Arbiter)&lt;/li&gt;
	&lt;li&gt;Data / Journal / Log are on separated disks, all mounted with LUKS filesystem&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I suspect LUKS to be the source of the problem.&lt;/p&gt;

&lt;p&gt;The running updates are using $addToSet / $elemMatch.&lt;/p&gt;

&lt;p&gt;I attached the error information and backtrace, let me know if you need more info.&lt;/p&gt;</description>
                <environment>Ubuntu Server 16.04 / LUKS filesystem</environment>
        <key id="356637">SERVER-28084</key>
            <summary>WT_PANIC: WiredTiger library panic during bulk update</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="mark.agarunov">Mark Agarunov</assignee>
                                    <reporter username="hmducoulombier@marketing1by1.com">Henri-Maxime Ducoulombier</reporter>
                        <labels>
                            <label>envns</label>
                            <label>rns</label>
                            <label>wtc</label>
                    </labels>
                <created>Thu, 23 Feb 2017 16:46:30 +0000</created>
                <updated>Fri, 27 Jul 2018 18:58:41 +0000</updated>
                            <resolved>Mon, 6 Mar 2017 21:03:25 +0000</resolved>
                                    <version>3.4.2</version>
                                                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1534788" author="hmducoulombier@marketing1by1.com" created="Tue, 28 Mar 2017 14:18:22 +0000"  >&lt;p&gt;Little follow up about this issue.&lt;/p&gt;

&lt;p&gt;We tweaked the server and downgraded the RAM frequency (it&apos;s 2400Mhz DDR3 RAM) because the Motherboard considers 1800+ frequency as overclocking, and it&apos;s working fine so far (a complete memtest helped us identify instability in the RAM).&lt;/p&gt;

&lt;p&gt;Thanks again for you time spent on investigating the issue.&lt;/p&gt;

&lt;p&gt;Henri-Maxime&lt;/p&gt;</comment>
                            <comment id="1517216" author="hmducoulombier@marketing1by1.com" created="Mon, 6 Mar 2017 21:14:33 +0000"  >&lt;p&gt;Hello @Mark Agarunov,&lt;/p&gt;

&lt;p&gt;Thanks for investigating. I&apos;m hoping to be able to build a new development server with more hardware (and not 3 volumes per disk) and to build the LUKS encrypted volumes over xfs and not ext4 filesystem this time (does this issue happen with not encrypted ext4 too ?).&lt;/p&gt;

&lt;p&gt;Can&apos;t do that right now, we would need to invest both in hardware and time to migrate the datas, but this should happen in 2017. If the problem still occurs then, I&apos;ll post a new bug report.&lt;/p&gt;

&lt;p&gt;I&apos;ve been looking of the web for info on LUKS and possible problems like that, but it seems that there are not so many people building systems like that, and literature is quite thin.&lt;/p&gt;

&lt;p&gt;Last but not least, the problem occurred today while only reading data, whereas it was occurring with massive updates/write before, this is why I thought it was important to post the logs this time.&lt;/p&gt;

&lt;p&gt;Thx&lt;br/&gt;
Henri-Maxime&lt;/p&gt;</comment>
                            <comment id="1517070" author="mark.agarunov" created="Mon, 6 Mar 2017 19:29:19 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=hmducoulombier%40marketing1by1.com&quot; class=&quot;user-hover&quot; rel=&quot;hmducoulombier@marketing1by1.com&quot;&gt;hmducoulombier@marketing1by1.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for providing these logs. Looking over the output, it appears that the corruption is outside of Mongodb. Errors of this type frequently indicate an underlying hardware/storage layer issue. The error messages indicate that the data has changed between when it was written by Mongodb and when it was read:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;read checksum error for 12288B block at offset 691863552: calculated block checksum of 3408525212 doesn&apos;t match expected checksum of 3304251198&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;You are correct that a repair may not be needed with journaling enabled to get MongoDB running again, it may be the case there is a previously corrupted portion of the database, and the errors only occur when it is accessed, not on startup. A repair may fix this issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1516631" author="hmducoulombier@marketing1by1.com" created="Mon, 6 Mar 2017 15:08:06 +0000"  >&lt;p&gt;Here is a recent backtrace (just happened now) while doing a &quot;simple&quot; aggregate on datas (same environment, 3 shards reading at the same time on the same disk)&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/150796/150796_jira-err.log&quot; title=&quot;jira-err.log attached to SERVER-28084&quot;&gt;jira-err.log&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Hope it helps.&lt;/p&gt;</comment>
                            <comment id="1507532" author="hmducoulombier@marketing1by1.com" created="Thu, 23 Feb 2017 19:51:37 +0000"  >&lt;p&gt;Another backtrace of another (yet similiar) issue.&lt;/p&gt;</comment>
                            <comment id="1507529" author="hmducoulombier@marketing1by1.com" created="Thu, 23 Feb 2017 19:51:12 +0000"  >&lt;p&gt;Hello Mark,&lt;/p&gt;

&lt;p&gt;Here are the answsers to your questions :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Only one of the &lt;tt&gt;mongod&lt;/tt&gt; crashes, sometimes 2 (but not the same, nor at the same time). Also, it&apos;s not necessarily the primary.&lt;/li&gt;
	&lt;li&gt;The storage are SSD directly on hardware. 1 SSD is for the data, 1 for the journals and 1 for the logs. Each disk is partitioned with 12 partitions, and each &lt;tt&gt;mongod&lt;/tt&gt; uses its own 3 partitions (I&apos;m including the config servers here).&lt;/li&gt;
	&lt;li&gt;We do not need to repair the database, restarting the affected mongod is enough (journaling is enabled, this is probably why). However, we did have another crash, more severe in my opinion, and had to restore the affected member of the replica set (from the secondary) because it had invalid BSON and it was making some requests impossible. I&apos;m attaching a second log with this other backtrace (repairing the member failed (no output at all))&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;I&apos;ll post the complete log as soon as I can.&lt;/p&gt;

&lt;p&gt;Thanks for investigating this issue.&lt;/p&gt;</comment>
                            <comment id="1507332" author="mark.agarunov" created="Thu, 23 Feb 2017 17:27:17 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=hmducoulombier%40marketing1by1.com&quot; class=&quot;user-hover&quot; rel=&quot;hmducoulombier@marketing1by1.com&quot;&gt;hmducoulombier@marketing1by1.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report. After looking over the output you&apos;ve provided, I have a few questions and requests so that we can better investigate this:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Do all of the &lt;tt&gt;mongod&lt;/tt&gt; instances crash or only once specific instance?&lt;/li&gt;
	&lt;li&gt;What is the layout of the underlying storage? (lvm, virtual block storage, directly on hardware, etc)&lt;/li&gt;
	&lt;li&gt;Have you tried &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/repairDatabase/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;running a repair&lt;/a&gt; on the affected database(s)?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Additionally, please send the complete logs from any affected &lt;tt&gt;mongod&lt;/tt&gt; instances by this behavior. &lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="150087" name="001.txt" size="7846" author="hmducoulombier@marketing1by1.com" created="Thu, 23 Feb 2017 19:51:22 +0000"/>
                            <attachment id="150072" name="002.txt" size="9982" author="hmducoulombier@marketing1by1.com" created="Thu, 23 Feb 2017 16:46:17 +0000"/>
                            <attachment id="150796" name="jira-err.log" size="9664058" author="hmducoulombier@marketing1by1.com" created="Mon, 6 Mar 2017 15:07:38 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 23 Feb 2017 17:27:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 46 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>nick.brewer</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 46 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>hmducoulombier@marketing1by1.com</customfieldvalue>
            <customfieldvalue>mark.agarunov</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht2uqn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsvefb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4fz3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>