<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:05:35 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4305] Deadlock of secondary trying to sync the oplog if index versions are mixed on master</title>
                <link>https://jira.mongodb.org/browse/SERVER-4305</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;What we want to do:&lt;br/&gt;
upgrade all indexes to version 2.0 (v:1) in our replicaset&lt;/p&gt;

&lt;p&gt;How we do this:&lt;br/&gt;
we start with one secondary, shut it down, change port and remove repset param, start it with repair command to reindex all indexes.&lt;br/&gt;
after repair reset configuration and connect it to the repset. Wait until the slave is up2date to proceed with the next secondary.&lt;/p&gt;

&lt;p&gt;What is the problem:&lt;br/&gt;
The secondary is not able to catch up with the master. It has a single process running with 100% cpu usage and almost idle io. (cpu bound)&lt;br/&gt;
It falls slowly more and more behind. (all hosts have the same hardware)&lt;/p&gt;


&lt;p&gt;What we suspect:&lt;br/&gt;
We have a database which has some indexes with the old version and some with the new. If a secondary upgrades the indexes, it has all indexes on the latest version and this locks the replay/resync of the oplog from the master which still has the mixed version indexes.&lt;/p&gt;

&lt;p&gt;We downgraded the indexes again with an older mongod binary (1.8.4). After this was finished, we connected the secondary to the replicaset again and it replayed the oplog without a problem and is now in sync again.&lt;/p&gt;

&lt;p&gt;All hosts have the mongod binary version 2.0.1.&lt;/p&gt;

&lt;p&gt;I&apos;ve attached iostat and mongostat output. The host with the problem is mn01.&lt;br/&gt;
In the mongod.log is no error message just some reoccurring message about the cursor:&lt;br/&gt;
Wed Nov 16 06:32:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;rsSync&amp;#93;&lt;/span&gt; repl: old cursor isDead, will initiate a new one&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Steffen&lt;/p&gt;
</description>
                <environment>Linux 2.6.32-35-server, Ubuntu 10.04, MongoDB 2.0.1, Replicaset with 3 Nodes, NUMA, 2x XEON E5620 , 24 GB RAM </environment>
        <key id="25049">SERVER-4305</key>
            <summary>Deadlock of secondary trying to sync the oplog if index versions are mixed on master</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.mongodb.org/images/icons/priorities/blocker.svg">Blocker - P1</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="steffen">Steffen</reporter>
                        <labels>
                            <label>index</label>
                            <label>replication</label>
                            <label>stale</label>
                    </labels>
                <created>Thu, 17 Nov 2011 16:53:35 +0000</created>
                <updated>Mon, 14 May 2012 20:38:51 +0000</updated>
                            <resolved>Mon, 14 May 2012 20:38:51 +0000</resolved>
                                    <version>2.0.1</version>
                                                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="119324" author="ian@10gen.com" created="Mon, 14 May 2012 20:38:51 +0000"  >&lt;p&gt;Closing as Cannot Reproduce - please reopen if you continue to run into the same problem.&lt;/p&gt;</comment>
                            <comment id="80760" author="steffen" created="Wed, 18 Jan 2012 18:17:36 +0000"  >&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;we tried another run to upgrade the indexes.&lt;br/&gt;
this time I used compact command on each collection in each database starting on the secondaries. Switched master to secondary and upgraded indexes there too.&lt;br/&gt;
All went fine so far. No problems with the oplog sync.&lt;/p&gt;

&lt;p&gt;Maybe -repair startup parameter doesn&apos;t work, but compact does.&lt;/p&gt;</comment>
                            <comment id="76074" author="steffen" created="Tue, 27 Dec 2011 11:39:26 +0000"  >&lt;p&gt;We will try another rebuild of the indexes next year. I will report then about the results.&lt;/p&gt;</comment>
                            <comment id="72840" author="kristina" created="Tue, 13 Dec 2011 19:26:39 +0000"  >&lt;p&gt;Sorry about the delay, consulting with coworkers.&lt;/p&gt;

&lt;p&gt;Can you attach gdb and get a stack trace for all threads at a few different times, so that we can see what code is running when the CPU is pegged?&lt;/p&gt;

&lt;p&gt;To get a backtrace of all threads in gdb, run:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;(gdb) thread apply all bt&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="70073" author="steffen" created="Thu, 1 Dec 2011 15:26:04 +0000"  >&lt;p&gt;Yes, already posted:&lt;br/&gt;
secondary-14-nov-repair.log.zip&lt;/p&gt;</comment>
                            <comment id="69915" author="kristina" created="Wed, 30 Nov 2011 20:35:03 +0000"  >&lt;p&gt;Unfortunately, MMS doesn&apos;t give us that kind of insight (it&apos;s just statistics about your data, not the data or queries themselves).&lt;/p&gt;

&lt;p&gt;That does look like something weird is happening during the repair, do you happen to have the log from when you did the repair?&lt;/p&gt;</comment>
                            <comment id="68509" author="steffen" created="Wed, 23 Nov 2011 10:10:51 +0000"  >&lt;p&gt;Can you read from mms the operations happening? &lt;/p&gt;

&lt;p&gt;The obvious difference between this two replicasets, we are using, is just the index versions of the databases.&lt;br/&gt;
Since we suspect the index version mix, I will show you the index output from the database which has both versions. Maybe in the repair something gets messed up. I hope you can reproduce the problem with this indexes.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;name&quot; : &quot;_id_&quot;, &quot;ns&quot; : &quot;exstream.objects&quot;, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;v&quot; : 0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;name&quot; : &quot;_id_&quot;, &quot;ns&quot; : &quot;exstream.connections&quot;, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;v&quot; : 0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;name&quot; : &quot;uniqueIndex&quot;, &quot;ns&quot; : &quot;exstream.connections&quot;, &quot;unique&quot; : true, &quot;dropDups&quot; : true, &quot;key&quot; : { &quot;parentId&quot; : 1, &quot;objectId&quot; : 1 }, &quot;v&quot; : 0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.index.subscription&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.filter.subscription&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.likes&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;parentId&quot; : 1, &quot;objectId&quot; : 1, &quot;created&quot; : 1, &quot;updated&quot; : 1 }, &quot;unique&quot; : true, &quot;ns&quot; : &quot;exstream.likes&quot;, &quot;name&quot; : &quot;uniqueIndex&quot;, &quot;dropDups&quot; : true }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.comments&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;parentId&quot; : 1, &quot;objectId&quot; : 1, &quot;created&quot; : 1, &quot;updated&quot; : 1 }, &quot;unique&quot; : true, &quot;ns&quot; : &quot;exstream.comments&quot;, &quot;name&quot; : &quot;uniqueIndex&quot;, &quot;dropDups&quot; : true }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;name&quot; : &quot;_id_&quot;, &quot;ns&quot; : &quot;exstream.filter.follow&quot;, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;v&quot; : 0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;name&quot; : &quot;_id_&quot;, &quot;ns&quot; : &quot;exstream.index.follow&quot;, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;v&quot; : 0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.connection.subscribe&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;s&quot; : 1, &quot;e&quot; : 1 }, &quot;unique&quot; : true, &quot;ns&quot; : &quot;exstream.connection.subscribe&quot;, &quot;name&quot; : &quot;uniqueIndex.connection&quot;, &quot;dropDups&quot; : true }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.connection.block&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;s&quot; : 1, &quot;e&quot; : 1 }, &quot;unique&quot; : true, &quot;ns&quot; : &quot;exstream.connection.block&quot;, &quot;name&quot; : &quot;uniqueIndex.connection&quot;, &quot;dropDups&quot; : true }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.kv.lastRead&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;key&quot; : 1 }, &quot;ns&quot; : &quot;exstream.kv.lastRead&quot;, &quot;name&quot; : &quot;key_1&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;_id&quot; : 1 }, &quot;ns&quot; : &quot;exstream.kv.count&quot;, &quot;name&quot; : &quot;_id_&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;key&quot; : 1 }, &quot;ns&quot; : &quot;exstream.kv.count&quot;, &quot;name&quot; : &quot;key_1&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;uid&quot; : 1, &quot;s&quot; : 1 }, &quot;ns&quot; : &quot;exstream.filter.follow&quot;, &quot;name&quot; : &quot;idx2&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;uid&quot; : 1, &quot;s&quot; : 1 }, &quot;ns&quot; : &quot;exstream.filter.subscription&quot;, &quot;name&quot; : &quot;idx2&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;uid&quot; : 1, &quot;c&quot; : 1, &quot;v&quot; : 1, &quot;s&quot; : 1 }, &quot;ns&quot; : &quot;exstream.filter.follow&quot;, &quot;name&quot; : &quot;idx1&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;uid&quot; : 1, &quot;c&quot; : 1, &quot;v&quot; : 1, &quot;s&quot; : 1 }, &quot;ns&quot; : &quot;exstream.filter.subscription&quot;, &quot;name&quot; : &quot;idx1&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;k&quot; : 1, &quot;p&quot; : 1, &quot;vi&quot; : 1, &quot;ac&quot; : 1 }, &quot;ns&quot; : &quot;exstream.index.follow&quot;, &quot;name&quot; : &quot;idx_asc&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;k&quot; : 1, &quot;p&quot; : -1, &quot;vi&quot; : 1, &quot;ac&quot; : 1 }, &quot;ns&quot; : &quot;exstream.index.follow&quot;, &quot;name&quot; : &quot;idx_desc&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;k&quot; : 1, &quot;p&quot; : 1, &quot;vi&quot; : 1, &quot;ac&quot; : 1 }, &quot;ns&quot; : &quot;exstream.index.subscription&quot;, &quot;name&quot; : &quot;idx_asc&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;v&quot; : 1, &quot;key&quot; : { &quot;k&quot; : 1, &quot;p&quot; : -1, &quot;vi&quot; : 1, &quot;ac&quot; : 1 }, &quot;ns&quot; : &quot;exstream.index.subscription&quot;, &quot;name&quot; : &quot;idx_desc&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="68393" author="kristina" created="Tue, 22 Nov 2011 22:01:22 +0000"  >&lt;p&gt;Okay, thanks.  I have been unable to reproduce so far. Can you describe what kinds of operations are happening on the primary?&lt;/p&gt;</comment>
                            <comment id="68175" author="steffen" created="Tue, 22 Nov 2011 08:39:27 +0000"  >&lt;p&gt;Should be the 15th.&lt;br/&gt;
Are you able to reproduce the problem?&lt;/p&gt;</comment>
                            <comment id="67992" author="kristina" created="Mon, 21 Nov 2011 17:51:05 +0000"  >&lt;p&gt;Was the mongostat taken on the 15th?  16th?&lt;/p&gt;</comment>
                            <comment id="67628" author="steffen" created="Fri, 18 Nov 2011 17:08:44 +0000"  >&lt;p&gt;log of repair process and first start after repair.&lt;/p&gt;</comment>
                            <comment id="67624" author="steffen" created="Fri, 18 Nov 2011 16:56:32 +0000"  >&lt;p&gt;Log of the secondary from nov 15&lt;/p&gt;</comment>
                            <comment id="67623" author="kristina" created="Fri, 18 Nov 2011 16:48:27 +0000"  >&lt;p&gt;Thanks for all the information.  I&apos;m not sure what day mongostat was from, but from MMS, it looks like it was up to November 15th that would be of interest, log-wise.  Do you have the secondary&apos;s log from then?  Or was the lock up after that spike?&lt;/p&gt;

&lt;p&gt;(Also, please don&apos;t mark comments viewable by Users only, it just messes us up (and they&apos;re still visible to everyone).)&lt;/p&gt;</comment>
                            <comment id="67424" author="steffen" created="Thu, 17 Nov 2011 17:43:44 +0000"  >&lt;p&gt;log of the secondary which is behind the master and tries to catch up.&lt;/p&gt;</comment>
                            <comment id="67423" author="steffen" created="Thu, 17 Nov 2011 17:42:53 +0000"  >&lt;p&gt;all data on this replicaset uses 125 GB.&lt;/p&gt;

&lt;p&gt;We have tried this with another node which did a full resync but we ran into this bug: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-4294&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-4294&lt;/a&gt;&lt;br/&gt;
If we don&apos;t hit the edge case the node also locks with 100% not catching up.&lt;/p&gt;

&lt;p&gt;We have another replicaset with all indexes at version 1.8 (v:0). We added there also a clean node for backup purposes which has the new indexes from the initial sync and it works without problems.&lt;/p&gt;</comment>
                            <comment id="67417" author="eliot" created="Thu, 17 Nov 2011 16:57:54 +0000"  >&lt;p&gt;Can you attach the log from the secondary when it can&apos;t keep up?&lt;/p&gt;

&lt;p&gt;How big is the data set?&lt;/p&gt;

&lt;p&gt;Have you tried not doing a repair, but just letting a secondary do a full resync?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="13863" name="iostat.txt" size="56233" author="steffen" created="Thu, 17 Nov 2011 16:53:36 +0000"/>
                            <attachment id="13862" name="mongostat.txt" size="197590" author="steffen" created="Thu, 17 Nov 2011 16:53:35 +0000"/>
                            <attachment id="13879" name="secondary-14-nov-repair.log.zip" size="3685683" author="steffen" created="Fri, 18 Nov 2011 17:08:44 +0000"/>
                            <attachment id="13878" name="secondary-15-nov.log.zip" size="63526" author="steffen" created="Fri, 18 Nov 2011 16:56:32 +0000"/>
                            <attachment id="13864" name="secondary.log.txt" size="94544" author="steffen" created="Thu, 17 Nov 2011 17:43:44 +0000"/>
                            <attachment id="13871" name="strace.txt" size="26516" author="steffen" created="Fri, 18 Nov 2011 10:11:15 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>16.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 17 Nov 2011 16:57:54 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        11 years, 40 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            11 years, 40 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>steffen</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hroldr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrit3r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23368</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlglb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>