<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:24:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-10905] Duplicate key error killed all secondaries on cluster</title>
                <link>https://jira.mongodb.org/browse/SERVER-10905</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a cluster of 3 nodes to enable replication. Two of the nodes are  configured with the same priority (1) while the third one is configured with priority 0 so it never gets promoted (it&apos;s the one we use for backups). &lt;/p&gt;

&lt;p&gt;As of last night, the two secondaries have been failing continuously during writes, with the following error logged on both servers:&lt;/p&gt;

&lt;p&gt;Wed Sep 25 05:28:16.211 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 2&amp;#93;&lt;/span&gt; ERROR: writer worker caught exception: E11000 duplicate key error index: marketshare.Application.$name_1  dup key: { : &quot;TestingRDS&quot; } on: { ts: Timestamp 1380112101000|1, h: -5880003146554201345, v: 2, op: &quot;u&quot;, ns: &quot;marketshare.Application&quot;, o2: &lt;/p&gt;
{ _id: ObjectId(&apos;5242be375f6ffb0c37145385&apos;) }
&lt;p&gt;, o: { $set: { boxes.0: &lt;/p&gt;
{ box_type_name: &quot;OPTIMIZERMS-APPV4&quot;, updated: &quot;2013-09-25 12:28:22.855168&quot;, &amp;lt;rest of object&amp;gt;...}
&lt;p&gt;}}&lt;/p&gt;

&lt;p&gt;Wed Sep 25 05:28:16.211 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 2&amp;#93;&lt;/span&gt;   Fatal Assertion 16360&lt;br/&gt;
0xdddd81 0xd9dc13 0xc26bfc 0xdab721 0xe26609 0x7ff9f4923e9a 0x7ff9f3c36ccd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdddd81&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9dc13&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc26bfc&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdab721&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe26609&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f4923e9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f3c36ccd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Wed Sep 25 05:28:16.215 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 2&amp;#93;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;***aborting after fassert() failure&lt;/p&gt;


&lt;p&gt;Wed Sep 25 05:28:16.215 Got signal: 6 (Aborted).&lt;/p&gt;

&lt;p&gt;Wed Sep 25 05:28:16.219 Backtrace:&lt;br/&gt;
0xdddd81 0x6d0d29 0x7ff9f3b794a0 0x7ff9f3b79425 0x7ff9f3b7cb8b 0xd9dc4e 0xc26bfc 0xdab721 0xe26609 0x7ff9f4923e9a 0x7ff9f3c36ccd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdddd81&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6d0d29&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f3b794a0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f3b79425&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f3b7cb8b&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9dc4e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc26bfc&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdab721&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe26609&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f4923e9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7ff9f3c36ccd&amp;#93;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;After rebooting both secondaries, the cluster was able to establish quorum again for about 15 minutes, but after that, the issue reproed again. When we logged in to the instances directly,  we noticed that que indexes were correct on the primary, but were completely empty on the secondary.&lt;/p&gt;</description>
                <environment>ubuntu 12.04 on aws</environment>
        <key id="91523">SERVER-10905</key>
            <summary>Duplicate key error killed all secondaries on cluster</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="samantha.ritter@mongodb.com">Samantha Ritter</assignee>
                                    <reporter username="raberrel">Ramiro Berrelleza</reporter>
                        <labels>
                    </labels>
                <created>Wed, 25 Sep 2013 18:17:55 +0000</created>
                <updated>Wed, 10 Dec 2014 23:18:58 +0000</updated>
                            <resolved>Tue, 18 Mar 2014 11:25:02 +0000</resolved>
                                    <version>2.4.6</version>
                                                    <component>Replication</component>
                                        <votes>1</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="518229" author="stennie" created="Tue, 18 Mar 2014 11:25:02 +0000"  >&lt;p&gt;Hi Ramiro,&lt;/p&gt;

&lt;p&gt;Please be advised I&apos;m now closing this issue. It looks like something went amiss in your replica set upgrade back in September, which you resolved by rebuilding the replica set on fresh servers. From the log files provided it is unclear how to reproduce this issue.&lt;/p&gt;

&lt;p&gt;If you do have any further questions or concerns on this issue, please let us know.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Stephen&lt;/p&gt;</comment>
                            <comment id="458046" author="raberrel" created="Mon, 18 Nov 2013 20:57:17 +0000"  >&lt;p&gt;No, we haven&apos;t seen the issue repro. I&apos;ve launched a parallel set of servers with the same topology, and haven&apos;t seen this issue. However, these are fresh servers, instead of upgraded ones, so it&apos;s not really the same scenario 100%.&lt;/p&gt;</comment>
                            <comment id="451181" author="samantha.ritter@10gen.com" created="Mon, 4 Nov 2013 21:17:58 +0000"  >&lt;p&gt;Hi Ramiro, I&apos;m sorry for the delay in response.  Has this issue reappeared, with either your original cluster or your parallel set of servers?&lt;/p&gt;</comment>
                            <comment id="432733" author="raberrel" created="Sat, 28 Sep 2013 00:24:45 +0000"  >&lt;p&gt;We removed the server from the cluster, so for now we&apos;re running in a single server until we understand the problem better. &lt;/p&gt;

&lt;p&gt;The history of the cluster goes as follows:&lt;br/&gt;
Originally, we had  a primary and a replica running on AWS (mongodb 2.2.3, on us-west-1). After about 3-4 months running on that setup, we decided to migrate our servers to run with a primary and two replicas (primary and 1 secondary on AWS, the other secondary on our datacenter, all mongodb 2.4.6). In order to migrate the data and avoid disruptions, we decided to join our new primary to the old setup, and once replication was complete, we promoted it to primary (by calling rs.stepDown() on the old server), removed the old servers from the configuration, and added the new servers to the replica set. Our servers ran fine for about a week, until the reported incident happened.&lt;/p&gt;

&lt;p&gt;We have a parallel set of servers with the same configuration, but with different data. We haven&apos;t seen the issue repro on those servers.&lt;/p&gt;</comment>
                            <comment id="432716" author="eliot" created="Fri, 27 Sep 2013 23:16:18 +0000"  >&lt;p&gt;What&apos;s the status now?&lt;/p&gt;

&lt;p&gt;What&apos;s the history of the cluster? i.e. restores from backup, syncs, etc...&lt;/p&gt;
</comment>
                    </comments>
                    <attachments>
                            <attachment id="31828" name="mongodb-primary.log" size="26136865" author="raberrel" created="Wed, 25 Sep 2013 18:17:55 +0000"/>
                            <attachment id="31839" name="mongodb-secondary-1.log" size="26108035" author="raberrel" created="Wed, 25 Sep 2013 18:51:37 +0000"/>
                            <attachment id="31840" name="mongodb-secondary-2.log" size="25913805" author="raberrel" created="Wed, 25 Sep 2013 18:51:37 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 27 Sep 2013 23:16:18 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 48 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 48 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>raberrel</customfieldvalue>
            <customfieldvalue>samantha.ritter@mongodb.com</customfieldvalue>
            <customfieldvalue>stephen.steneker@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrmf3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrwlen:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>104202</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsbyxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>