<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:19:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-9059] Fatal Assertion on replication</title>
                <link>https://jira.mongodb.org/browse/SERVER-9059</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi, I&apos;m having a hell of a time getting a new replica set member added.&lt;/p&gt;

&lt;p&gt;Brand new machine with mongodb 2.4 and tried to add a new member to my replica set. Got a SEGFAULT.&lt;/p&gt;

&lt;p&gt;I have no idea what&apos;s causing this issue, but I got the exact same segfault a few months ago when 2.2 came out. I repaired the db and STILL got this problem. And it only happens after the entire replication is finished (6 hours later).&lt;/p&gt;

&lt;p&gt;I can&apos;t make any new copies of the data! And I need to start sharding because my service is exploding in growth. What am I supposed to do if I can&apos;t replicate?&lt;/p&gt;

&lt;p&gt;Sam&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;/p&gt;

&lt;p&gt;Thu Mar 21 02:01:21.926 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt; build index done.  scanned 9 total records. 0.001 secs&lt;br/&gt;
Thu Mar 21 02:01:22.040 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; warning replset primary is an older version than we are; upgrade recommended&lt;br/&gt;
Thu Mar 21 02:01:25.229 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt; ERROR: exception: E11000 duplicate key error index: newsblur.stories.$story_hash_1  dup key: { : &quot;980568:65f04f&quot; } on: { ts: Timestamp 1363810091000|32, h: -7663483269809030053, op: &quot;i&quot;, ns: &quot;newsblur.stories&quot;, o: { _id: ObjectId(&apos;514a172a0f55a7137913adea&apos;), share_user_ids: {}, story_author_name: &quot;&quot;, story_date: new Date(1363809963000), story_tags: {}, story_hash: &quot;980568:65f04f&quot;, story_feed_id: 980568, story_permalink: &quot;http://feedproxy.google.com/~r/DrudgeReportFeed/~3/Kn0_WSFPhAI/us-syria-crisis-lebanon-idUSBRE92J0AY20130320&quot;, story_title: &quot;More Syrian shells land in Lebanon...&quot;, comment_user_ids: {}, story_content_z: BinData, story_guid: &quot;http://www.reuters.com/article/2013/03/20/us-syria-crisis-lebanon-idUSBRE92J0AY20130320&quot; } }&lt;br/&gt;
Thu Mar 21 02:01:25.230 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt;   Fatal Assertion 16361&lt;br/&gt;
0xdcae01 0xd8ab83 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd &lt;br/&gt;
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcae01&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd8ab83&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ecb6&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd986b1&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe13709&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fac48fe9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fab7a2cbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Thu Mar 21 02:01:25.238 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;***aborting after fassert() failure&lt;/p&gt;


&lt;p&gt;Thu Mar 21 02:01:25.238 Got signal: 6 (Aborted).&lt;/p&gt;

&lt;p&gt;Thu Mar 21 02:01:25.242 Backtrace:&lt;br/&gt;
0xdcae01 0x6ce879 0x7f1fab6e54a0 0x7f1fab6e5425 0x7f1fab6e8b8b 0xd8abbe 0xc1ecb6 0xd986b1 0xe13709 0x7f1fac48fe9a 0x7f1fab7a2cbd &lt;br/&gt;
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcae01&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6ce879&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fab6e54a0&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fab6e5425&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fab6e8b8b&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd8abbe&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo7replset21multiInitialSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x166) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ecb6&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd986b1&amp;#93;&lt;/span&gt;&lt;br/&gt;
/usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe13709&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fac48fe9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f1fab7a2cbd&amp;#93;&lt;/span&gt;&lt;/p&gt;</description>
                <environment>Ubuntu 64-bit 12.04, 32GB on Digital Ocean</environment>
        <key id="69273">SERVER-9059</key>
            <summary>Fatal Assertion on replication</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="rassi">J Rassi</assignee>
                                    <reporter username="samuel@ofbrooklyn.com">Samuel Clay</reporter>
                        <labels>
                            <label>initialSync</label>
                            <label>replication</label>
                    </labels>
                <created>Thu, 21 Mar 2013 17:08:40 +0000</created>
                <updated>Tue, 16 Nov 2021 15:07:43 +0000</updated>
                            <resolved>Mon, 25 Mar 2013 18:05:44 +0000</resolved>
                                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="330544" author="rassi@10gen.com" created="Tue, 7 May 2013 21:05:27 +0000"  >&lt;blockquote&gt;
&lt;p&gt;I can still create a new ticket, but I won&apos;t be bale to provide much but the broken server&apos;s log file.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;That would be a fine start.&lt;/p&gt;

&lt;p&gt;If more convenient than uploading to JIRA, you may provide your logs via scp.  Use the following command (hit enter at the password prompt):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;scp -P 722 &amp;lt;file(s)&amp;gt; newsblur@www.10gen.com:&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="330526" author="samuel@ofbrooklyn.com" created="Tue, 7 May 2013 20:48:49 +0000"  >&lt;p&gt;The problem is that I don&apos;t have an easy way to get your logs for a few days ago. The logs are enormous, as they have millions of &quot;connections&quot; lines. I can still create a new ticket, but I won&apos;t be bale to provide much but the broken server&apos;s log file.&lt;/p&gt;</comment>
                            <comment id="330340" author="rassi@10gen.com" created="Tue, 7 May 2013 16:25:27 +0000"  >&lt;p&gt;Sam, I don&apos;t see any new issues created from your account.  Could we still help you diagnose this?&lt;/p&gt;</comment>
                            <comment id="328868" author="rassi@10gen.com" created="Sun, 5 May 2013 17:55:08 +0000"  >&lt;p&gt;I believe this is unrelated to the original problem encountered above.  Would you open a new ticket for this issue?  Please include the mongod log for db24 and db25 covering at least the last 24 hours, and the mongod log for db22 covering up to the last two weeks if available.  Thanks.&lt;/p&gt;</comment>
                            <comment id="328604" author="samuel@ofbrooklyn.com" created="Sun, 5 May 2013 03:51:36 +0000"  >&lt;p&gt;I&apos;m getting a new assertion failure on replication. This time after I created a new index on primary and had it replicate to the secondaries. After the secondaries finished, 2/3 of them came up just fine. One machine did now and it gave a stack trace. The broken machine is db22.newsblur.com (on MMS), and all machines are running 2.4.3.&lt;/p&gt;

&lt;p&gt;&amp;#8211;&lt;/p&gt;

&lt;p&gt;Sun May  5 03:47:43.376 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet still syncing, not yet to minValid optime 5185cf69:1&lt;br/&gt;
Sun May  5 03:47:43.524 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet SECONDARY&lt;br/&gt;
Sun May  5 03:47:43.525 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl prefetch worker&amp;#93;&lt;/span&gt;  newsblur.stories Assertion failure _a != -1 src/mongo/db/ops/../pdfile.h 607&lt;br/&gt;
Sun May  5 03:47:43.525 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl prefetch worker&amp;#93;&lt;/span&gt;  newsblur.stories Assertion failure _a != -1 src/mongo/db/ops/../pdfile.h 607&lt;br/&gt;
0xdcf361 0xd902bd 0x9cdf38 0x9d6dbf 0xb2c0b4 0xb2cf55 0xc1ca02 0xd9cc01 0xe17cb9 0x7f22bb81be9a 0x7f22bab2bcbd &lt;br/&gt;
Sun May  5 03:47:43.525 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl prefetch worker&amp;#93;&lt;/span&gt;  newsblur.stories Assertion failure _a != -1 src/mongo/db/ops/../pdfile.h 607&lt;br/&gt;
0xdcf3610xdcf361 0xd902bd  0xd902bd0x9cdf38  0x9cdf38 103172470x9d6dbf  0xb2c0b40xb2c0b4  0xb2cf55 0xb2cf550xc1ca02  0xd9cc01 0xe17cb90xc1ca02  0xd9cc010x7f22bb81be9a  0xe17cb90x7f22bab2bcbd  0x7f22bb81be9a &lt;br/&gt;
0x7f22bab2bcbd &lt;br/&gt;
Sun May  5 03:47:43.528 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl prefetch worker&amp;#93;&lt;/span&gt;  newsblur.stories Assertion failure _a != -1 src/mongo/db/ops/../pdfile.h 607&lt;br/&gt;
0xdcf361 0xd902bd 0x9cdf38 0x9d6dbf 0xb2c0b4 0xb2cf55 0xc1ca02 0xd9cc01 0xe17cb9 0x7f22bb81be9a 0x7f22bab2bcbd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd902bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0x118) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9cdf38&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d6dbf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2c0b4&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2cf55&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ca02&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd902bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0x118) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9cdf38&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d6dbf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2c0b4&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2cf55&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ca02&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd902bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0x118) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9cdf38&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d6dbf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2c0b4&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2cf55&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ca02&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd902bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0x118) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9cdf38&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d6dbf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2c0b4&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xb2cf55&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1ca02&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Sun May  5 03:47:43.552 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt;  newsblur.stories Assertion failure _a != -1 src/mongo/db/ops/../pdfile.h 607&lt;br/&gt;
0xdcf361 0xd902bd 0x9cdf38 0x9d6dbf 0x9d8c22 0xac0009 0xac130f 0xa8b72a 0xa8d787 0xa6956a 0xc1fb13 0xc1f258 0xd9cc01 0xe17cb9 0x7f22bb81be9a 0x7f22bab2bcbd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd902bd&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0x118) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9cdf38&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d6dbf&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo24indexRecordUsingTwoStepsEPKcPNS_16NamespaceDetailsENS_7BSONObjENS_7DiskLocEb+0x2d2) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x9d8c22&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1239) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xac0009&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xac130f&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2efa) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa8b72a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa8d787&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x65a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xa6956a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1fb13&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1f258&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Sun May  5 03:47:43.562 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt; ERROR: writer worker caught exception: assertion src/mongo/db/ops/../pdfile.h:607 on: { ts: Timestamp 1367723893000|1, h: 256075113332974832, v: 2, op: &quot;i&quot;, ns: &quot;newsblur.stories&quot;, o: { _id: ObjectId(&apos;5185cf75409b612f1631dce1&apos;), share_user_ids: {}, story_author_name: &quot;&quot;, story_date: new Date(1367723893192), story_tags: {}, story_hash: &quot;6:388392&quot;, story_feed_id: 6, story_permalink: &quot;http://lcamtuf.blogspot.com/2013/05/some-harmless-old-fashioned-fun-with-css.html&quot;, story_title: &quot;Some harmless, old-fashioned fun with CSS&quot;, comment_user_ids: {}, story_content_z: BinData, story_guid: &quot;http://lcamtuf.blogspot.com/2013/05/some-harmless-old-fashioned-fun-with-css.html&quot; } }&lt;br/&gt;
Sun May  5 03:47:43.562 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt;   Fatal Assertion 16360&lt;br/&gt;
0xdcf361 0xd8f0d3 0xc1f33c 0xd9cc01 0xe17cb9 0x7f22bb81be9a 0x7f22bab2bcbd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd8f0d3&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1f33c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;br/&gt;
Sun May  5 03:47:43.566 &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 1&amp;#93;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;***aborting after fassert() failure&lt;/p&gt;


&lt;p&gt;Sun May  5 03:47:43.566 Got signal: 6 (Aborted).&lt;/p&gt;

&lt;p&gt;Sun May  5 03:47:43.568 Backtrace:&lt;br/&gt;
0xdcf361 0x6cf729 0x7f22baa6e4a0 0x7f22baa6e425 0x7f22baa71b8b 0xd8f10e 0xc1f33c 0xd9cc01 0xe17cb9 0x7f22bb81be9a 0x7f22bab2bcbd &lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xdcf361&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x6cf729&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22baa6e4a0&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22baa6e425&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22baa71b8b&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd8f10e&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xc1f33c&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xd9cc01&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0xe17cb9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bb81be9a&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7f22bab2bcbd&amp;#93;&lt;/span&gt;&lt;/p&gt;

</comment>
                            <comment id="296128" author="rassi@10gen.com" created="Fri, 22 Mar 2013 19:54:24 +0000"  >&lt;p&gt;We have identified the root cause of the index lookup failure after the initial sync succeeded.  The details of the problem can be found in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-9087&quot; title=&quot;Index changes lost during initial sync period&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-9087&quot;&gt;&lt;del&gt;SERVER-9087&lt;/del&gt;&lt;/a&gt;, as well as a plan for a release with the fix.&lt;/p&gt;

&lt;p&gt;In short, you will need to run mongod with --repair on any 2.4.0 nodes that have done an initial sync, to fix the indexing issue.  The repair will rebuild the indexes from the clean data, which was not affected by the problem.&lt;/p&gt;

&lt;p&gt;If you&apos;d like any help with this process, please feel free to call us (646-201-9247), or feel free to email me a phone number to reach you at.&lt;/p&gt;</comment>
                            <comment id="295987" author="rassi@10gen.com" created="Fri, 22 Mar 2013 17:27:25 +0000"  >&lt;p&gt;As an aside, make sure you&apos;re running with an &lt;a href=&quot;http://docs.mongodb.org/manual/administration/replica-set-architectures/#sets-with-four-or-more-members&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;an odd number of members&lt;/a&gt; (if you&apos;re still on four, you&apos;ll want to &lt;a href=&quot;http://docs.mongodb.org/manual/administration/replica-sets/#replica-set-procedure-add-arbiter&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;add an arbiter&lt;/a&gt;).&lt;/p&gt;</comment>
                            <comment id="295926" author="rassi@10gen.com" created="Fri, 22 Mar 2013 16:27:01 +0000"  >&lt;p&gt;Could you connect to db21 and run the following (to increase the verbosity of the log output):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({setParameter:1, logLevel:2})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Please post when you next see that assertion (you can reset it back to 0 afterwards).&lt;/p&gt;</comment>
                            <comment id="295916" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 16:17:47 +0000"  >&lt;p&gt;Yes, the cluster is stable. All four nodes (the 3 secondaries) are replicating. But I&apos;m getting a few of these assertions. It seems they aren&apos;t taking down the server, though.&lt;/p&gt;</comment>
                            <comment id="295911" author="rassi@10gen.com" created="Fri, 22 Mar 2013 16:12:12 +0000"  >&lt;p&gt;Looking into it.&lt;/p&gt;

&lt;p&gt;Is db21 still replicating?  And db05/db20?&lt;/p&gt;</comment>
                            <comment id="295878" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 15:43:27 +0000"  >&lt;p&gt;This server is called db21, syncing from the same db04/db05 (primary/secondary).&lt;/p&gt;</comment>
                            <comment id="295876" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 15:42:46 +0000"  >&lt;p&gt;Bad news, added another member to the replica set and got a different assertion error after replicating. See attached mongodb_3rd_try.txt.&lt;/p&gt;

&lt;p&gt;Assertion: 10334:BSONObj size: -286331154 (0xEEEEEEEE) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: ObjectId(&apos;514abf3b8210885b8cdd197a&apos;)&lt;/p&gt;</comment>
                            <comment id="295812" author="rassi@10gen.com" created="Fri, 22 Mar 2013 14:22:27 +0000"  >&lt;p&gt;Glad the node is back up and running.  Closing this ticket; feel free to re-open if you have further questions related to this.&lt;/p&gt;</comment>
                            <comment id="295596" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 06:46:56 +0000"  >&lt;p&gt;Thank you guys, everything worked.&lt;/p&gt;</comment>
                            <comment id="295507" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 03:05:48 +0000"  >&lt;p&gt;Yup, that&apos;s exactly what I did. I&apos;m now 40 minutes into a 3 hour replication. (Took 6 hours the first time pre-repair, now takes 3 hours.)&lt;/p&gt;</comment>
                            <comment id="295505" author="rassi@10gen.com" created="Fri, 22 Mar 2013 02:59:21 +0000"  >&lt;p&gt;This is a known issue that still affects the current version, unfortunately.  I can suggest an ugly workaround, however: there&apos;s a ~3 second window that the new member will be in state STARTUP2, during which it has a copy of the replication configuration, before it has chosen a sync target.  During this window you can run rs.syncFrom().&lt;/p&gt;

&lt;p&gt;So: run rs.add(&apos;db20:27017&apos;) on db04, watch the mongod output on db20, and then as you see the line &quot;&lt;span class=&quot;error&quot;&gt;&amp;#91;rsStart&amp;#93;&lt;/span&gt; replSet STARTUP2&quot;, run rs.syncFrom(&apos;db05:27017&apos;) on db20 from your other shell.  It&apos;s okay if you run the syncFrom too early &amp;#8211; just try again.&lt;/p&gt;

&lt;p&gt;If you&apos;re successful, you should see the line &quot;&lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; replSet syncing to: db05:27017 by request&quot; in the log soon after.&lt;/p&gt;</comment>
                            <comment id="295487" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 02:27:59 +0000"  >&lt;p&gt;Now that I wiped the data dir, how do I use rs.syncFrom(&apos;db05:27017&apos;)? If I use it on db20, I get this message:&lt;/p&gt;

    {
	&quot;startupStatus&quot; : 3,
	&quot;info&quot; : &quot;run rs.initiate(...) if not yet done for the set&quot;,
	&quot;ok&quot; : 0,
	&quot;errmsg&quot; : &quot;can&apos;t get local.system.replset config from self or any seed (EMPTYCONFIG)&quot;
    }</comment>
                            <comment id="295482" author="rassi@10gen.com" created="Fri, 22 Mar 2013 02:22:41 +0000"  >&lt;p&gt;Yes, please retry your initial sync using an empty directory as the dbpath.&lt;/p&gt;

&lt;p&gt;The server does wipe user data before performing an initial sync (and no, there&apos;s no way to perform an initial sync that skips this step), but it does not drop the &quot;local&quot; database, which contains the oplog.  I believe that the oplog on db20 contains entries generated from a 2.2.0 mongod (the log snippet that you uploaded hints at this), which would result in you running into the linked issue even when running 2.4.0.  If you perform a clean 2.4.0 =&amp;gt; 2.4.0 initial sync, then the only oplog entries that will be applied will be those that occurred since the start of the initial sync, which will necessarily include no 2.2.0 oplog entries.&lt;/p&gt;</comment>
                            <comment id="295471" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 01:51:13 +0000"  >&lt;p&gt;No, mongodb deleted everything (so I think). Should I try again? The data is all there on db20, is there any way to salvage what&apos;s already there? This takes 5+ hours each time.&lt;/p&gt;</comment>
                            <comment id="295468" author="rassi@10gen.com" created="Fri, 22 Mar 2013 01:48:59 +0000"  >&lt;p&gt;When you tried performing the initial sync using 2.4.0, did you start mongod with a completely empty directory as the dbpath?&lt;/p&gt;</comment>
                            <comment id="295450" author="rassi@10gen.com" created="Fri, 22 Mar 2013 01:17:13 +0000"  >&lt;p&gt;Sent &amp;#8211; feel free to reply with host details in the email, but we&apos;ll leave this the main channel for the issue, assuming that&apos;s okay with you.&lt;/p&gt;</comment>
                            <comment id="295439" author="samuel@ofbrooklyn.com" created="Fri, 22 Mar 2013 00:56:02 +0000"  >&lt;p&gt;They are heavily used servers. The logs are huge, but they are filled to the brim with connection/end connection statements, as well as insertion/deletions/getmores.&lt;/p&gt;

&lt;p&gt;If you want to ssh in to read on the logs, it&apos;s 91GB (whoops, forget to initiate logrotate.d), just send me your ssh key. samuel@newsblur.com&lt;/p&gt;</comment>
                            <comment id="295437" author="rassi@10gen.com" created="Fri, 22 Mar 2013 00:51:21 +0000"  >&lt;p&gt;Can you post logs for db04 and db05?  As much as you have from today (and other runs of db20, if you have).&lt;/p&gt;</comment>
                            <comment id="295417" author="rassi@10gen.com" created="Fri, 22 Mar 2013 00:25:56 +0000"  >&lt;p&gt;We&apos;re taking a look at the log.&lt;/p&gt;</comment>
                            <comment id="295398" author="samuel@ofbrooklyn.com" created="Thu, 21 Mar 2013 23:50:26 +0000"  >&lt;p&gt;Bad news. Same segfault. See attached &quot;mongodb_2nd_try.txt&quot;. And just to confirm, every member of the set is running mongodb 2.4. And I just repaired the primary database yesterday.&lt;/p&gt;</comment>
                            <comment id="295285" author="rassi@10gen.com" created="Thu, 21 Mar 2013 21:13:00 +0000"  >&lt;p&gt;Can you upload the current mongod log for db20, so we can double-check what&apos;s going on?&lt;/p&gt;</comment>
                            <comment id="295277" author="samuel@ofbrooklyn.com" created="Thu, 21 Mar 2013 21:05:18 +0000"  >&lt;p&gt;Nevermind, found the syncFrom command (&lt;a href=&quot;http://docs.mongodb.org/manual/reference/method/rs.syncFrom/#rs.syncFrom&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/reference/method/rs.syncFrom/#rs.syncFrom&lt;/a&gt;). It was a bit tricky to figure out but I went with this method:&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Run rs.syncFrom(&apos;db05:27017&apos;) on db20 (forcing sync from secondary)&lt;/li&gt;
	&lt;li&gt;Run rs.add(&apos;db20:27017&apos;) on db04, which db20 is waiting in the syncFrom call.&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Replication has started. I&apos;ll check back here in 6 hours. Thanks!&lt;/p&gt;</comment>
                            <comment id="295245" author="samuel@ofbrooklyn.com" created="Thu, 21 Mar 2013 20:36:37 +0000"  >&lt;p&gt;Yes, that is all correct. I&apos;ll give it another try immediately. How can I force sync from another node?&lt;/p&gt;</comment>
                            <comment id="295222" author="rassi@10gen.com" created="Thu, 21 Mar 2013 20:02:03 +0000"  >&lt;p&gt;Can you confirm the following list of events:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;db04 and db05 were running mongod 2.2.0&lt;/li&gt;
	&lt;li&gt;db20 was brought up with mongod 2.4.0, which attempted to perform an initial sync&lt;/li&gt;
	&lt;li&gt;db20&apos;s initial sync failed&lt;/li&gt;
	&lt;li&gt;db04 and db05 were upgraded to mongod 2.4.0&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Assuming that this was the case, then you are likely encountering &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6671&quot; title=&quot;oplog is not strictly idempotent when unique index is present&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6671&quot;&gt;&lt;del&gt;SERVER-6671&lt;/del&gt;&lt;/a&gt;, which affects 2.2.0 (and was fixed in 2.2.1).  Since your cluster is now fully upgraded to 2.4.0, then you won&apos;t run into this issue during the initial sync.&lt;/p&gt;

&lt;p&gt;If you would like help syncing from another node or avoiding an initial sync, let us know.&lt;/p&gt;</comment>
                            <comment id="295209" author="samuel@ofbrooklyn.com" created="Thu, 21 Mar 2013 19:43:49 +0000"  >&lt;p&gt;They were running 2.2, but they are now both on 2.4. I can&apos;t perform a replication again without waiting 6 hours and severely impacting performance. There used to be an option to choose which db to perform the initial sync from, but that option is gone. If I could choose to use the secondary db, I wouldn&apos;t have as big of a performance hit, but my primary is on faster harder, so it gets selected by default.&lt;/p&gt;</comment>
                            <comment id="295207" author="rassi@10gen.com" created="Thu, 21 Mar 2013 19:38:07 +0000"  >&lt;p&gt;What version of mongod is db04 and db05 running?&lt;/p&gt;</comment>
                            <comment id="295084" author="samuel@ofbrooklyn.com" created="Thu, 21 Mar 2013 17:44:00 +0000"  >&lt;p&gt;nbset:PRIMARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
	&quot;set&quot; : &quot;nbset&quot;,&lt;br/&gt;
	&quot;date&quot; : ISODate(&quot;2013-03-21T17:43:20Z&quot;),&lt;br/&gt;
	&quot;myState&quot; : 1,&lt;br/&gt;
	&quot;members&quot; : [&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 2,&lt;br/&gt;
			&quot;name&quot; : &quot;db05:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 2,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;SECONDARY&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 43549,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1363887799,
				&quot;i&quot; : 99
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2013-03-21T17:43:19Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2013-03-21T17:43:19Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeatRecv&quot; : ISODate(&quot;2013-03-21T17:43:20Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 0&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 8,&lt;br/&gt;
			&quot;name&quot; : &quot;db04:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 1,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;PRIMARY&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 43728,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1363887800,
				&quot;i&quot; : 72
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2013-03-21T17:43:20Z&quot;),&lt;br/&gt;
			&quot;self&quot; : true&lt;br/&gt;
		}&lt;br/&gt;
	],&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;</comment>
                            <comment id="295077" author="rassi@10gen.com" created="Thu, 21 Mar 2013 17:38:04 +0000"  >&lt;p&gt;Hi Sam,&lt;/p&gt;

&lt;p&gt;I&apos;d like to collect more information to help diagnose this issue:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;How many members are in your replica set?  What version of mongod is each running?&lt;/li&gt;
	&lt;li&gt;Could you paste the output of running &lt;tt&gt;rs.status()&lt;/tt&gt; on the primary?&lt;/li&gt;
	&lt;li&gt;Could you upload the full mongod log for the member that threw the fatal assertion, as an attachment to this ticket?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;~ Jason Rassi&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="69437">SERVER-9087</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="45896">SERVER-6671</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="25250" name="mongodb.log" size="163591" author="samuelclay" created="Thu, 21 Mar 2013 17:49:31 +0000"/>
                            <attachment id="25269" name="mongodb_2nd_try.txt" size="93283" author="samuelclay" created="Thu, 21 Mar 2013 23:50:48 +0000"/>
                            <attachment id="25288" name="mongodb_3rd_try.txt" size="307505" author="samuelclay" created="Fri, 22 Mar 2013 15:43:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>33.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 21 Mar 2013 17:38:04 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 41 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 41 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>rassi</customfieldvalue>
            <customfieldvalue>samuel@ofbrooklyn.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrn0lj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrmunr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>47056</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;rs.add(&apos;db03:27017&apos;)&lt;/p&gt;

&lt;p&gt;...&lt;/p&gt;

&lt;p&gt;&amp;lt;wait 6 hours for 100GB of data to replicate&amp;gt;&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrsppr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>