<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:01:52 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3031] Replication fails with &quot;objects in a capped ns cannot grow&quot;</title>
                <link>https://jira.mongodb.org/browse/SERVER-3031</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;I have two secondaries, both of which stopped replication at around the same time with the following messages:&lt;/p&gt;

&lt;p&gt;----------------------------------------------------------------------------------------------------------------------------------------------&lt;br/&gt;
Secondary 1:&lt;br/&gt;
Mon May  2 02:10:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 13106 nextSafe(): &lt;/p&gt;
{ $err: &quot;capped cursor overrun during query: local.oplog.rs&quot;, code: 13338 }
&lt;p&gt;Mon May  2 02:11:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
.....&lt;br/&gt;
Mon May  2 08:24:56 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Mon May  2 08:25:07 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet error RS102 too stale to catch up, at least from ip-10-84-41-152:27017&lt;br/&gt;
---------------------------------------------------------------------------------------------------------------------------------------------&lt;br/&gt;
Secondary 2:&lt;br/&gt;
Mon May  2 02:10:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Mon May  2 02:10:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Mon May  2 02:11:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
....&lt;br/&gt;
Mon May  2 05:11:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 1000 replSet source for syncing doesn&apos;t seem to be await capable &amp;#8211; is it an older version of mongodb?&lt;br/&gt;
....&lt;br/&gt;
Mon May  2 08:24:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Mon May  2 08:25:09 &lt;span class=&quot;error&quot;&gt;&amp;#91;ReplSetHealthPollTask&amp;#93;&lt;/span&gt; replSet member ip-10-116-71-169 RECOVERING&lt;br/&gt;
---------------------------------------------------------------------------------------------------------------------------------------------&lt;/p&gt;

&lt;p&gt;I didn&apos;t have any alerts setup on these, and this caused the replicas to go stale and I am now getting them back up with a full resync, etc.&lt;/p&gt;

&lt;p&gt;I use a lot of capped collections in my code, but I don&apos;t update any of them in place. This looks like a MongoDB oplog capped collection issue.&lt;/p&gt;

&lt;p&gt;Any help would be appreciated.&lt;/p&gt;</description>
                <environment>Linux Kernel 2.6.32-309-ec2, x86_64, EC2 m2.2xlarge with Raid0 created using mdadm on EBS volumes of 1TB each.</environment>
        <key id="16546">SERVER-3031</key>
            <summary>Replication fails with &quot;objects in a capped ns cannot grow&quot;</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="nadahalli">Tejaswi Nadahalli</reporter>
                        <labels>
                    </labels>
                <created>Mon, 2 May 2011 15:30:06 +0000</created>
                <updated>Tue, 12 Jul 2016 00:19:11 +0000</updated>
                            <resolved>Thu, 5 May 2011 19:48:16 +0000</resolved>
                                    <version>1.8.1</version>
                                    <fixVersion>1.9.1</fixVersion>
                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="35172" author="eliot" created="Mon, 30 May 2011 14:43:22 +0000"  >&lt;p&gt;drop collection&lt;/p&gt;</comment>
                            <comment id="35160" author="nadahalli" created="Mon, 30 May 2011 11:55:19 +0000"  >&lt;p&gt;Can you elaborate on that?&lt;/p&gt;

&lt;p&gt;2 - Drop == drop collection? drop DB? drop Indexes?&lt;/p&gt;

&lt;p&gt;And why are are mongodump and mongorestore processes required?&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="35126" author="eliot" created="Mon, 30 May 2011 05:45:49 +0000"  >&lt;p&gt;Dropping and re-creating is best.&lt;br/&gt;
You can do:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;mongodump&lt;/li&gt;
	&lt;li&gt;drop&lt;/li&gt;
	&lt;li&gt;recreated&lt;/li&gt;
	&lt;li&gt;reindex&lt;/li&gt;
	&lt;li&gt;mongorestore&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="35106" author="nadahalli" created="Mon, 30 May 2011 03:21:08 +0000"  >&lt;p&gt;Well, the problem is _id indexes, if they exist, cannot be dropped.&lt;/p&gt;

&lt;p&gt;Is there a way to convert each _id index in the capped collection to be Unique? &lt;/p&gt;

&lt;p&gt;Or is my only option to delete all capped collections and re-create them with a unique _id index?&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="35098" author="eliot" created="Mon, 30 May 2011 02:07:02 +0000"  >&lt;p&gt;You need to make the _id index unique on the master capped collection.&lt;br/&gt;
Can you drop it and re add it as unique.&lt;/p&gt;</comment>
                            <comment id="35093" author="nadahalli" created="Sun, 29 May 2011 23:57:34 +0000"  >&lt;p&gt;Replication stopped again for me in production.&lt;/p&gt;

&lt;p&gt;I have indexes on _id on ALL my tables. It&apos;s still stopping replication with the same bug. Is it possible that the bug has something to do with the oplog capped collection and not my own DB capped collections?&lt;/p&gt;

&lt;p&gt;This is getting to a point now that having replica sets is completely useless. I sync from scratch every 3 days or so.&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34589" author="nadahalli" created="Thu, 26 May 2011 16:49:56 +0000"  >&lt;p&gt;You can see that the unique index could be subsumed by the search index. Below, I am creating indexes on two separate capped collections. One has unique:true, and the other doesn&apos;t. But showIndexes doesn&apos;t distinguish between them. &lt;br/&gt;
-----------------------------------------------------------------------------------------------------------&lt;br/&gt;
test_mongo:PRIMARY&amp;gt; db.coll_3.ensureIndex({_id:1}, &lt;/p&gt;
{unique:true}
&lt;p&gt;)&lt;br/&gt;
test_mongo:PRIMARY&amp;gt; db.coll_3.getIndexes()&lt;br/&gt;
[&lt;br/&gt;
	{&lt;br/&gt;
		&quot;name&quot; : &quot;&lt;em&gt;id&lt;/em&gt;&quot;,&lt;br/&gt;
		&quot;ns&quot; : &quot;test.coll_3&quot;,&lt;br/&gt;
		&quot;key&quot; : &lt;/p&gt;
{
			&quot;_id&quot; : 1
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;v&quot; : 0&lt;br/&gt;
	}&lt;br/&gt;
]&lt;br/&gt;
-----------------------------------------------------------------------------------------------------------&lt;/p&gt;

&lt;p&gt;test_mongo:PRIMARY&amp;gt; db.coll_4.ensureIndex({_id:1})                          &lt;br/&gt;
test_mongo:PRIMARY&amp;gt; db.coll_4.getIndexes()                                  &lt;br/&gt;
[&lt;br/&gt;
	{&lt;br/&gt;
		&quot;name&quot; : &quot;&lt;em&gt;id&lt;/em&gt;&quot;,&lt;br/&gt;
		&quot;ns&quot; : &quot;test.coll_4&quot;,&lt;br/&gt;
		&quot;key&quot; : &lt;/p&gt;
{
			&quot;_id&quot; : 1
		}
&lt;p&gt;,&lt;br/&gt;
		&quot;v&quot; : 0&lt;br/&gt;
	}&lt;br/&gt;
]&lt;br/&gt;
-----------------------------------------------------------------------------------------------------------&lt;br/&gt;
So, given that I have a regular _id index on ALL my capped collections, do I have to now create a unique index on all of them? Or does that come for free?&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34587" author="kristina" created="Thu, 26 May 2011 16:42:02 +0000"  >&lt;p&gt;Creating an index builds a tree of values for a given key, so that you can find a value (or range of values) in ~O(log&lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/thumbs_down.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;) time instead of O&lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/thumbs_down.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt; (looking through every field of every record).  When you create a unique index, every value for a key must be distinct, so whenever you do an insert, the db checks to make sure that _id hasn&apos;t been used before in that collection.  Does that make sense?&lt;/p&gt;

&lt;p&gt;MongoDB automatically creates a unique _id index on non-capped collections, but does not for capped ones.&lt;/p&gt;</comment>
                            <comment id="34581" author="nadahalli" created="Thu, 26 May 2011 16:27:51 +0000"  >&lt;p&gt;No, my question was slightly different.&lt;/p&gt;

&lt;p&gt;What&apos;s the difference between a unique index and a regular vanilla index on the _id key with no other fields involved?&lt;/p&gt;

&lt;p&gt;As in, what&apos;s the difference between db.collection.ensureIndex({_id:1}, &lt;/p&gt;
{unique=true}
&lt;p&gt;) and db.collection.ensureIndex({_id:1}})?&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34550" author="kristina" created="Thu, 26 May 2011 14:53:51 +0000"  >&lt;p&gt;If you have a unique index on _id, the script you gave above (to reproduce the replication halt) won&apos;t work.  Maybe you created a non-unique _id index or something on one of your capped collections?&lt;/p&gt;</comment>
                            <comment id="34410" author="nadahalli" created="Wed, 25 May 2011 19:46:59 +0000"  >&lt;p&gt;Are _id indexes &apos;unique&apos; by default?&lt;/p&gt;

&lt;p&gt;I am using &lt;a href=&quot;http://pastebin.com/5EBRhZKc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastebin.com/5EBRhZKc&lt;/a&gt; to figure out what tables to index.&lt;/p&gt;

&lt;p&gt;Every collection in my database has an _id index. &lt;/p&gt;

&lt;p&gt;What gives?&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34372" author="nadahalli" created="Wed, 25 May 2011 16:53:37 +0000"  >&lt;p&gt;Ok. Thanks for the input. Let me see if I can create these indexes on all my capped collections on the primary, replicate from scratch, and see if it works.&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34369" author="kristina" created="Wed, 25 May 2011 16:46:41 +0000"  >&lt;p&gt;The problem is that, if you&apos;re going to use _id in a capped collection, you need to create a unique index on the _id field (because it isn&apos;t done automatically for you with capped collections).  I&apos;ll add some documentation about it, but this is a weird &quot;works as intended&quot; edge case.&lt;/p&gt;</comment>
                            <comment id="34333" author="kristina" created="Wed, 25 May 2011 14:58:06 +0000"  >&lt;p&gt;Perfect, thanks!  I&apos;ll let you know.&lt;/p&gt;</comment>
                            <comment id="34331" author="nadahalli" created="Wed, 25 May 2011 14:52:59 +0000"  >&lt;p&gt;Ok. Done with simulation. Here&apos;re the required steps:&lt;/p&gt;

&lt;p&gt;mongod --dbpath 1 --logpath 1/mongodb.log --fork --replSet test_mongo --oplogSize 1000 --port 27001&lt;br/&gt;
mongod --dbpath 2 --logpath 2/mongodb.log --fork --replSet test_mongo --oplogSize 1000 --port 27002&lt;br/&gt;
mongod --dbpath 3 --logpath 3/mongodb.log --fork --replSet test_mongo --oplogSize 1000 --port 27003&lt;/p&gt;

&lt;p&gt;mongo --port 27001&lt;/p&gt;

&lt;p&gt;rs.initiate()&lt;br/&gt;
rs.add(&apos;secondary1&apos;)&lt;br/&gt;
rs.add(&apos;secondary2&apos;)&lt;/p&gt;

&lt;p&gt;Then, run this &lt;a href=&quot;http://pastebin.com/1J78nxNv&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastebin.com/1J78nxNv&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;All the above code does it creates a capped collection and inserts two documents into the capped collection with the same _id, but with differing document sizes. It works in the primary because there is no default _id for capped collections in the primary. On the secondaries it fails because IIRC, the secondaries create an index on _id to aid replication. This _id doesn&apos;t allow the document in the replicated capped collection to grow. But the key thing is, this shouldn&apos;t stop replication from continuing with other documents.&lt;/p&gt;

&lt;p&gt;I have a feeling this might not be an easy fix - at least, removing the _id dependency from capped collection replication might be hard. But the second part of the bug - where replication stops happening if the first bug is encountered, might be easy to fix. Just speculating here &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34330" author="kristina" created="Wed, 25 May 2011 14:48:34 +0000"  >&lt;p&gt;I think that something that is an insert on the primary may be being turned into an upsert in the oplog (which happens sometimes and is correct, but it looks like there&apos;s an edge case that we haven&apos;t handled).  Are you MapReducing into any capped collections?&lt;/p&gt;</comment>
                            <comment id="34323" author="nadahalli" created="Wed, 25 May 2011 14:31:16 +0000"  >&lt;p&gt;I am still working on reproducing it in a non-production setup. Might take a while.&lt;/p&gt;

&lt;p&gt;As food for thought though, even if there is some code that is causing a capped collection object to increase in size on the primary, that should not stop replication to the secondaries, right? Wouldn&apos;t just that behaviour be a good starting point to investigate this? I am just thinking out loud here &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Will get a reproducable setup though.&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34015" author="nadahalli" created="Mon, 23 May 2011 21:09:45 +0000"  >&lt;p&gt;I have a hunch on how I can locally replicate the setup that reproduces the bug. Will try to do that.&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="34011" author="kristina" created="Mon, 23 May 2011 20:40:03 +0000"  >&lt;p&gt;A copy of the oplog.rs document that&apos;s causing the issue would probably allow me to solve it.  When this happens, can you connect to the secondary (that has stopped replicating) and run:&lt;/p&gt;

&lt;p&gt;&amp;gt; use local&lt;br/&gt;
&amp;gt; db.getMongo().setSlaveOk()&lt;br/&gt;
&amp;gt; doc = db.oplog.rs.find().sort(&lt;/p&gt;
{natural:-1}
&lt;p&gt;).limit(1).next()&lt;/p&gt;

&lt;p&gt;Then connect to the primary and find a document greater than the last one on the secondary?  I think this should do it:&lt;/p&gt;

&lt;p&gt;&amp;gt; use local&lt;br/&gt;
&amp;gt; db.oplog.rs.find({ts : {$gt : new Timestamp(doc.ts.t, doc.ts.i)}}).limit(1)&lt;/p&gt;

&lt;p&gt;Or if you know the code that is doing this, send exactly what it is sending to the db?&lt;/p&gt;</comment>
                            <comment id="33882" author="nadahalli" created="Mon, 23 May 2011 06:17:04 +0000"  >&lt;p&gt;This bug is driving me crazy &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Anyway, happened again with this logged in the primary: &lt;/p&gt;

&lt;p&gt;Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn36600&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1250 126ms&lt;br/&gt;
Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn38285&amp;#93;&lt;/span&gt; Assertion: 13338:capped cursor overrun during query: local.oplog.rs&lt;br/&gt;
Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn38285&amp;#93;&lt;/span&gt; assertion 13338 capped cursor overrun during query: local.oplog.rs ns:local.oplog.rs query:{}&lt;br/&gt;
Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn38285&amp;#93;&lt;/span&gt;  ntoskip:0 ntoreturn:1&lt;br/&gt;
Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn38285&amp;#93;&lt;/span&gt; query local.oplog.rs ntoreturn:1 exception  452ms&lt;/p&gt;

&lt;p&gt;The secondary logged this:&lt;/p&gt;

&lt;p&gt;Sun May 22 02:20:41 &lt;span class=&quot;error&quot;&gt;&amp;#91;FileAllocator&amp;#93;&lt;/span&gt; done allocating datafile /raiddrive/master/stream_db_p.143, size: 2047MB,  took 0.023 secs&lt;br/&gt;
Sun May 22 13:51:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;br/&gt;
Sun May 22 13:51:13 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 13106 nextSafe(): &lt;/p&gt;
{ $err: &quot;capped cursor overrun during query: local.oplog.rs&quot;, code: 13338 }
&lt;p&gt;Sun May 22 13:51:25 &lt;span class=&quot;error&quot;&gt;&amp;#91;replica set sync&amp;#93;&lt;/span&gt; replSet syncThread: 10003 failing update: objects in a capped ns cannot grow&lt;/p&gt;

&lt;p&gt;I am still not on the 1.9 series to get the debug messages that you added. I can do that if you think it&apos;ll definitely help. Also, Skot and I did some analysis the other day, and I did notice that there was some application code that was inadvertently increasing the size of an object in a capped collection. This was due to duplication in _id creation. (I don&apos;t use _id in my own code, and rely on the default creation to just work). Is it possible that I am being hit by 2 bugs here - one to do with _id creation and how sometimes duplicates get created during inserts to capped collections and two - that when this error happens, slaves stop replication.&lt;/p&gt;

&lt;p&gt;Any help would be appreciated.&lt;/p&gt;

&lt;p&gt;-T&lt;/p&gt;</comment>
                            <comment id="31728" author="kristina" created="Mon, 9 May 2011 21:29:12 +0000"  >&lt;p&gt;That would be great (or if you can reproduce).&lt;/p&gt;</comment>
                            <comment id="31726" author="nadahalli" created="Mon, 9 May 2011 21:24:42 +0000"  >&lt;p&gt;Same bug strikes again. Both my stales have gone stale because of the oplog error. My master has only 6 or so hours of oplog and that got overwritten.&lt;/p&gt;

&lt;p&gt;I will try to set alerts so that if this happens again, I can look at the master oplog immediately.&lt;/p&gt;</comment>
                            <comment id="30821" author="auto" created="Tue, 3 May 2011 15:58:56 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;login&apos;: u&apos;kchodorow&apos;, u&apos;name&apos;: u&apos;Kristina&apos;, u&apos;email&apos;: u&apos;kristina@10gen.com&apos;}
&lt;p&gt;Message: more debugging info on sync errors &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3031&quot; title=&quot;Replication fails with &amp;quot;objects in a capped ns cannot grow&amp;quot;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3031&quot;&gt;&lt;del&gt;SERVER-3031&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/5eaaddba616acf65c55bb4c0ca0b7f9c48f27999&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/5eaaddba616acf65c55bb4c0ca0b7f9c48f27999&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="30820" author="kristina" created="Tue, 3 May 2011 15:58:22 +0000"  >&lt;p&gt;Hmm, I don&apos;t think so.  It seems like there was a bad operation in the oplog, but it&apos;ll be gone now.  I&apos;m going to add some better logging so that this is easier to debug in the future.&lt;/p&gt;</comment>
                            <comment id="30814" author="nadahalli" created="Tue, 3 May 2011 15:25:13 +0000"  >&lt;p&gt;No, that would have rolled over &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/sad.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;db.oplog.rs.find({}, &lt;/p&gt;
{&quot;ts&quot;:1}
&lt;p&gt;).sort({$natural:-1}).limit(1)&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1304436284000, &quot;i&quot; : 551 }
&lt;p&gt; }&lt;/p&gt;

&lt;p&gt;db.oplog.rs.find({}, &lt;/p&gt;
{&quot;ts&quot;:1}
&lt;p&gt;).sort({$natural:1}).limit(1)&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1304413752000, &quot;i&quot; : 2455 }
&lt;p&gt; }&lt;/p&gt;

&lt;p&gt;I have around 6 hours worth primary&apos;s oplog, and as it rolled over, my secondaries went stale, I guess.&lt;/p&gt;

&lt;p&gt;I will be increasing the oplog size to around 2-3 days in the near future, fwiw &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;

&lt;p&gt;Is there anything else I can do?&lt;/p&gt;</comment>
                            <comment id="30809" author="kristina" created="Tue, 3 May 2011 15:19:41 +0000"  >&lt;p&gt;No, the primary&apos;s oplog: local.oplog.rs (if you have a backup or it hasn&apos;t rolled over).&lt;/p&gt;</comment>
                            <comment id="30804" author="nadahalli" created="Tue, 3 May 2011 15:13:00 +0000"  >&lt;p&gt;The Primary Log at the relevant point is:&lt;/p&gt;

&lt;p&gt;--------------------------------------------------------------------------------------------------------------&lt;br/&gt;
Mon May  2 02:10:21 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3923&amp;#93;&lt;/span&gt; getmore local.oplog.rs cid:8139024901366363643 getMore: { ts: &lt;/p&gt;
{ $gte: new Date(5597974341116494213) }
&lt;p&gt; }  bytes:4194710 nreturned:10690 4517ms&lt;br/&gt;
Mon May  2 02:10:21 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3922&amp;#93;&lt;/span&gt; getmore local.oplog.rs cid:8804929124998056910 getMore: { ts: &lt;/p&gt;
{ $gte: new Date(5597974341116494213) }
&lt;p&gt; }  bytes:4194659 nreturned:8256 3660ms&lt;br/&gt;
Mon May  2 02:10:30 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2989&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1237 163ms&lt;br/&gt;
Mon May  2 02:10:34 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2989&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1237 231ms&lt;br/&gt;
Mon May  2 02:10:36 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2989&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1237 759ms&lt;br/&gt;
Mon May  2 02:10:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3923&amp;#93;&lt;/span&gt; end connection 10.116.71.169:55883&lt;br/&gt;
Mon May  2 02:10:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3922&amp;#93;&lt;/span&gt; end connection 10.119.54.196:52978&lt;br/&gt;
Mon May  2 02:10:48 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2989&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1237 656ms&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.116.71.169:36468 #15823&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2989&amp;#93;&lt;/span&gt; query admin.$cmd ntoreturn:1 command: &lt;/p&gt;
{ serverStatus: 1 }
&lt;p&gt; reslen:1237 103ms&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15823&amp;#93;&lt;/span&gt; Assertion: 13338:capped cursor overrun during query: local.oplog.rs&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15823&amp;#93;&lt;/span&gt; assertion 13338 capped cursor overrun during query: local.oplog.rs ns:local.oplog.rs query:{}&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15823&amp;#93;&lt;/span&gt;  ntoskip:0 ntoreturn:1&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15823&amp;#93;&lt;/span&gt; query local.oplog.rs ntoreturn:1 exception  299ms&lt;br/&gt;
Mon May  2 02:10:50 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15823&amp;#93;&lt;/span&gt; end connection 10.116.71.169:36468&lt;br/&gt;
Mon May  2 02:10:51 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.119.54.196:37407 #15824&lt;br/&gt;
Mon May  2 02:10:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15824&amp;#93;&lt;/span&gt; query local.oplog.rs reslen:99167 nscanned:103 { ts: &lt;/p&gt;
{ $gte: new Date(5601935417654903123) }
&lt;p&gt; }  nreturned:101 445ms&lt;br/&gt;
Mon May  2 02:10:52 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15824&amp;#93;&lt;/span&gt; end connection 10.119.54.196:37407&lt;br/&gt;
Mon May  2 02:11:01 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.116.71.169:36469 #15825&lt;br/&gt;
Mon May  2 02:11:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15825&amp;#93;&lt;/span&gt; query local.oplog.rs reslen:99167 nscanned:103 { ts: &lt;/p&gt;
{ $gte: new Date(5601935417654903123) }
&lt;p&gt; }  nreturned:101 869ms&lt;br/&gt;
Mon May  2 02:11:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn15825&amp;#93;&lt;/span&gt; end connection 10.116.71.169:36469&lt;br/&gt;
--------------------------------------------------------------------------------------------------------------&lt;/p&gt;</comment>
                            <comment id="30779" author="kristina" created="Tue, 3 May 2011 13:00:55 +0000"  >&lt;p&gt;No thanks, I&apos;m interested in what operation the secondaries got stuck at (which would be in the primary&apos;s oplog).  If you have a copy of the primary&apos;s oplog from when 10003 started appearing, that would be very helpful.&lt;/p&gt;</comment>
                            <comment id="30743" author="nadahalli" created="Tue, 3 May 2011 01:58:21 +0000"  >&lt;p&gt;I cannot send that right now, as I have restarted both my stale secondaries and they are back syncing from the primary.&lt;/p&gt;

&lt;p&gt;But I did check it multiple times back when they had gone stale, and both the secondaries had the rs102 &quot;Too stale to catch up&quot; error.&lt;/p&gt;

&lt;p&gt;I have the server log and data from both the secondaries though, in a separate backup, in case you want to look at them, or want me to run some queries on them.&lt;/p&gt;</comment>
                            <comment id="30717" author="kristina" created="Mon, 2 May 2011 22:43:53 +0000"  >&lt;p&gt;Can you sent the output of running rs.status() on the primary?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>30.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 2 May 2011 22:43:53 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 38 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 38 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>nadahalli</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrp0kv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrif5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>21108</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrok2v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>