<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:19:58 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-29124] Fatal Assertion 16360</title>
                <link>https://jira.mongodb.org/browse/SERVER-29124</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;A replica set secondary crashed with the following assertion error:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-05-11T09:05:17.741-0400 F REPL     [repl writer worker 3] writer worker caught exception:  :: caused by :: 11000 E11000 duplicate key error &amp;lt;redacted&amp;gt;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-05-11T09:05:17.745-0400 I -        [repl writer worker 3] Fatal Assertion 16360&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-05-11T09:05:17.745-0400 I -        [repl writer worker 3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after fassert() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;At the time the primary was under load, servicing 3.6k reads and 132 writes per second.  The other replica set secondary was being rebuilt at the time this one crashed.&lt;/p&gt;

&lt;p&gt;There are multple &apos;duplicate key&apos; errors in the primary&apos;s log, where documents were rejected on insert, but none are the document reported as duplicate by the secondary that crashed.&lt;/p&gt;

&lt;p&gt;MongoDB 3.2.11 on Amazon Linux, version 2106.03&lt;br/&gt;
mongo setup:  three node replica set, one primary, two secondaries&lt;/p&gt;</description>
                <environment></environment>
        <key id="382723">SERVER-29124</key>
            <summary>Fatal Assertion 16360</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="blunney">Bob Lunney</reporter>
                        <labels>
                    </labels>
                <created>Thu, 11 May 2017 14:40:51 +0000</created>
                <updated>Fri, 9 Feb 2018 18:51:49 +0000</updated>
                            <resolved>Thu, 18 Jan 2018 22:06:44 +0000</resolved>
                                                                    <component>Index Maintenance</component>
                    <component>Replication</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1673950" author="blunney" created="Fri, 15 Sep 2017 15:09:14 +0000"  >&lt;p&gt;Kelsey,&lt;/p&gt;

&lt;p&gt;This issue has not recurred since the initial report.  We have also not had a failover since then, but we&apos;re not looking forward to the inevitable, either.&lt;/p&gt;

&lt;p&gt;Unfortunately, as a Mongo noob, I wasn&apos;t aware of the value of the &lt;tt&gt;diagnostics&lt;/tt&gt; directory, and probably destroyed any chance of solving this mystery.  I am aware that unique indexes on secondaries somehow relies on WiredTiger&apos;s MVCC mechanism, as dumps made from secondaries with &lt;tt&gt;mongodump&lt;/tt&gt; will sometimes have duplicate data that prevents unique index creation.&lt;/p&gt;

&lt;p&gt;If there is anything else I can do to help please let me know.  Otherwise I suggest closing the ticket as unsolvable, since the diagnostic data isn&apos;t available.&lt;/p&gt;

&lt;p&gt;Thanks for your efforts!&lt;/p&gt;

&lt;p&gt;Bob&lt;/p&gt;</comment>
                            <comment id="1673563" author="thomas.schubert" created="Fri, 15 Sep 2017 04:48:35 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blunney&quot; class=&quot;user-hover&quot; rel=&quot;blunney&quot;&gt;blunney&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We&apos;ve been working to understand what has happened here, but haven&apos;t had much success. Have you encountered this issue since the initial report?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1570242" author="blunney" created="Fri, 12 May 2017 21:23:32 +0000"  >&lt;p&gt;Thomas,&lt;/p&gt;

&lt;p&gt;Thanks for your help.&lt;/p&gt;

&lt;p&gt;I have uploaded :&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;rs0-mongo.log.gz&lt;/li&gt;
	&lt;li&gt;rs1-mongo.log.gz (see lines 1106 - 1110 for the error)&lt;/li&gt;
	&lt;li&gt;rs2-mongo.log.gz&lt;/li&gt;
	&lt;li&gt;rs2-diagnostic.data.tar.gz&lt;/li&gt;
	&lt;li&gt;rs2-indexes.txt&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Sadly, I don&apos;t have the &lt;tt&gt;diagnostic.data&lt;/tt&gt; files, nor the indexes for the affected collection from the secondaries at the time of the incident.  We needed the secondaries back, so the data directory was purged and the secondaries resynced with the primary.  I&apos;ll know better next time.  &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;The failed secondary (rs1) was the primary prior to the fatal assertion error.  The new primary (rs2) took over via an automatic failover.  Just prior to the automatic failover the other secondary (rs0) was shutdown, data directory purged, and restarted to resync it with the primary (rs1 at the time).  Then the failover event occurred, rs2 was elected primary, rs1 crashed, and eventually rs0 began resyncing from rs2.  &lt;/p&gt;

&lt;p&gt;At this point we have rs2 in PRIMARY mode, rs0 in STARTUP2, and rs1 down, i.e. no secondary to fail over to.  I let rs0 finish resyncing and transition to SECONDARY mode prior to resyncing rs1.&lt;/p&gt;

&lt;p&gt;Thanks for your help, and please let me know if there is anymore information I can provide.&lt;/p&gt;</comment>
                            <comment id="1570010" author="thomas.schubert" created="Fri, 12 May 2017 19:07:46 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blunney&quot; class=&quot;user-hover&quot; rel=&quot;blunney&quot;&gt;blunney&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/bc1bcea1-3734-4479-bb6f-20d23a41b872.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; where you can diagnostic files. Files uploaded to this portal are only visible to MongoDB employees investigating this issue and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;To help us investigate this issue, would you please provide the following information?&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;An archive of the diagnostic.data directory of the primary&lt;/li&gt;
	&lt;li&gt;An archive of the diagnostic.data directory of the affected secondary&lt;/li&gt;
	&lt;li&gt;Complete log files of the primary&lt;/li&gt;
	&lt;li&gt;Complete log files of the secondary&lt;/li&gt;
	&lt;li&gt;Output of db.collection.getIndexes() where collection is the affected collection executed against the primary&lt;/li&gt;
	&lt;li&gt;Output of db.collection.getIndexes() where collection is the affected collection executed against the affected secondary&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Thank you for your help,&lt;br/&gt;
Thomas&lt;/p&gt;
</comment>
                            <comment id="1569006" author="blunney" created="Thu, 11 May 2017 14:44:09 +0000"  >&lt;p&gt;Correction:  MongoDB 3.2.11 on Amazon Linux, version 2016.03&lt;/p&gt;

&lt;p&gt;Addition:  Running WiredTiger as the storage engine.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 12 May 2017 19:07:46 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 21 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>backlog-server-pm</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 21 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>blunney</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht7ad3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hszgrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Run mongod with replication.&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs0kin:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>