<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:05:25 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4248] Replica Set gets out of sync</title>
                <link>https://jira.mongodb.org/browse/SERVER-4248</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have seen an issue a few times on the same replica where the secondary is doing about 30-40k updates/sec and it has the state secondary still but can never catch up to master it just seems to stay the same amount behind master (normally just a few minutes).&lt;/p&gt;

&lt;p&gt;The primary has about 10 or less updates/sec so it doesn&apos;t seem to be an issue immediately of it doing too much for secondary to keep up.&lt;/p&gt;

&lt;p&gt;It has been fixed twice by stoping and starting the secondary and it will actually catch up after that normally.&lt;/p&gt;

&lt;p&gt;Also, when it happens the db.printReplicationInfo() will show a very small window of oplog time (&amp;lt; 30 min). The weird thing about it is that as time goes on if you keep checking that status the window is constantly growing but the first event time stays the same until the oplog grows large enough. So it is almost as if the oplog gets reset.&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt; db.printReplicationInfo()&lt;br/&gt;
configured oplog size:   4096MB&lt;br/&gt;
log length start to end: 92276secs (25.63hrs)&lt;br/&gt;
oplog first event time:  Wed Nov 09 2011 19:13:39 GMT+0000 (UTC)&lt;br/&gt;
oplog last event time:   Thu Nov 10 2011 20:51:35 GMT+0000 (UTC)&lt;br/&gt;
now:                     Thu Nov 10 2011 20:51:35 GMT+0000 (UTC)&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt; db.printReplicationInfo()&lt;br/&gt;
configured oplog size:   4096MB&lt;br/&gt;
log length start to end: 92488secs (25.69hrs)&lt;br/&gt;
oplog first event time:  Wed Nov 09 2011 19:13:39 GMT+0000 (UTC)&lt;br/&gt;
oplog last event time:   Thu Nov 10 2011 20:55:07 GMT+0000 (UTC)&lt;br/&gt;
now:                     Thu Nov 10 2011 20:55:07 GMT+0000 (UTC)&lt;/p&gt;</description>
                <environment>Ubuntu 11.04, AWS m2.2xlarge</environment>
        <key id="24764">SERVER-4248</key>
            <summary>Replica Set gets out of sync</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="benwyrosdick">Ben Wyrosdick</reporter>
                        <labels>
                            <label>oplog</label>
                            <label>replication</label>
                    </labels>
                <created>Thu, 10 Nov 2011 21:02:36 +0000</created>
                <updated>Mon, 11 Jul 2016 18:32:35 +0000</updated>
                            <resolved>Fri, 11 Nov 2011 03:51:10 +0000</resolved>
                                    <version>2.0.1</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>3</watches>
                                                                                                                <comments>
                            <comment id="66443" author="scotthernandez" created="Mon, 14 Nov 2011 01:20:13 +0000"  >&lt;p&gt;That is expected. Every update is done by _id in the oplog and they are idempotent. You will get a fan-out like this for any multi-operation from the primary.&lt;/p&gt;</comment>
                            <comment id="66442" author="benwyrosdick" created="Mon, 14 Nov 2011 01:08:36 +0000"  >&lt;p&gt;I think we see the op that causes it. It looks like maybe a single query is translated into thousands of ops in the oplog and causes it to get behind. Is this expected?&lt;/p&gt;

&lt;p&gt;== Ruby Snippet ==&lt;br/&gt;
User.collection.update({}, {&apos;$pullAll&apos; =&amp;gt; {unread_chats: &lt;span class=&quot;error&quot;&gt;&amp;#91;id&amp;#93;&lt;/span&gt;}}, &lt;/p&gt;
{multi: true}
&lt;p&gt;)&lt;/p&gt;

&lt;p&gt;== Oplog Afterwards ==&lt;br/&gt;
PRIMARY&amp;gt; db.oplog.rs.find()&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28384 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;7367950549734090104&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e4&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28385 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-4941163361012609057&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e5&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28386 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-5476149061517939211&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e6&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28387 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;2149060704232662888&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e7&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28388 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-5960082855673500958&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e8&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28389 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-5660342605049353919&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1e9&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28390 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-2221591118617756225&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1ea&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28391 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;479178575086387690&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1eb&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28392 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;1454718424901059596&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1ec&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28393 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-2733836735551696413&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1ed&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28394 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;3406322233193789257&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1ee&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28395 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;8502987010612628908&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1ef&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28396 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-4418903082715263754&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f0&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28397 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-4547385937031701051&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f1&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28398 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;4469202679615264403&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f2&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28399 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-4092244317294379346&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f3&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28400 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;3498665780605362592&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f4&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28401 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-7208861788328173337&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f5&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28402 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;-1232670496862447715&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f6&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;br/&gt;
{ &quot;ts&quot; : &lt;/p&gt;
{ &quot;t&quot; : 1321230472000, &quot;i&quot; : 28403 }
&lt;p&gt;, &quot;h&quot; : NumberLong(&quot;7642858452721797360&quot;), &quot;op&quot; : &quot;u&quot;, &quot;ns&quot; : &quot;xxxx_production.users&quot;, &quot;o2&quot; : &lt;/p&gt;
{ &quot;_id&quot; : ObjectId(&quot;4e9521f6722f7f0006e4e1f7&quot;) }
&lt;p&gt;, &quot;o&quot; : { &quot;$pullAll&quot; : &lt;/p&gt;
{ &quot;unread_chats&quot; : [ ObjectId(&quot;4ebfa07565ee19070b007eb6&quot;) ] }
&lt;p&gt; } }&lt;/p&gt;</comment>
                            <comment id="66104" author="eliot" created="Fri, 11 Nov 2011 03:51:10 +0000"  >&lt;p&gt;resolving as main issue seems done&lt;/p&gt;</comment>
                            <comment id="66103" author="eliot" created="Fri, 11 Nov 2011 03:48:47 +0000"  >&lt;p&gt;Yes&lt;/p&gt;

&lt;p&gt;use local&lt;br/&gt;
db.oplog.rs.find()&lt;/p&gt;</comment>
                            <comment id="66094" author="benwyrosdick" created="Fri, 11 Nov 2011 02:42:10 +0000"  >&lt;p&gt;can I query the oplog to see what the queries look like easily?&lt;/p&gt;</comment>
                            <comment id="66093" author="benwyrosdick" created="Fri, 11 Nov 2011 02:41:03 +0000"  >&lt;p&gt;yeah ... I agree after seeing it happen on staging.  The thing that was throwing me off was that the oplog start time seemed to hang for hours and I didn&apos;t think that it was just so many writes at once that it didn&apos;t move.&lt;/p&gt;

&lt;p&gt;I will increase the oplog size.&lt;/p&gt;</comment>
                            <comment id="66087" author="eliot" created="Fri, 11 Nov 2011 02:10:51 +0000"  >&lt;p&gt;Sounds like:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;write storm happens&lt;/li&gt;
	&lt;li&gt;time in oplog gets very low&lt;/li&gt;
	&lt;li&gt;secondary falls behind for a while&lt;/li&gt;
	&lt;li&gt;then catches up and oplog gets long again?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If that&apos;s the case, seems like pretty normal behavior.&lt;/p&gt;</comment>
                            <comment id="66071" author="benwyrosdick" created="Fri, 11 Nov 2011 01:00:51 +0000"  >&lt;p&gt;just to note ... the occurrence I posted about on staging acted a bit differently in that the first event in the oplog didn&apos;t stay the same like it had been on the production box, but having run the printReplicationInfo again on production just now it seems to have moved 4 seconds in the past few hours. This might possibly continue to point to some sort of huge burst in the oplog writes.&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt;  db.printReplicationInfo()&lt;br/&gt;
configured oplog size:   4096MB&lt;br/&gt;
log length start to end: 107032secs (29.73hrs)&lt;br/&gt;
oplog first event time:  Wed Nov 09 2011 19:13:43 GMT+0000 (UTC)&lt;br/&gt;
oplog last event time:   Fri Nov 11 2011 00:57:35 GMT+0000 (UTC)&lt;br/&gt;
now:                     Fri Nov 11 2011 00:57:35 GMT+0000 (UTC)&lt;/p&gt;</comment>
                            <comment id="66069" author="benwyrosdick" created="Fri, 11 Nov 2011 00:43:58 +0000"  >&lt;p&gt;It stays secondary ... it is actually acting up on one right now.&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt; db.printReplicationInfo()&lt;br/&gt;
configured oplog size:   4096MB&lt;br/&gt;
log length start to end: 1158secs (0.32hrs)&lt;br/&gt;
oplog first event time:  Fri Nov 11 2011 00:04:33 GMT+0000 (UTC)&lt;br/&gt;
oplog last event time:   Fri Nov 11 2011 00:23:51 GMT+0000 (UTC)&lt;br/&gt;
now:                     Fri Nov 11 2011 00:23:51 GMT+0000 (UTC)&lt;/p&gt;

&lt;p&gt;SECONDARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
	&quot;set&quot; : &quot;xxxx&quot;,&lt;br/&gt;
	&quot;date&quot; : ISODate(&quot;2011-11-11T00:19:25Z&quot;),&lt;br/&gt;
	&quot;myState&quot; : 2,&lt;br/&gt;
	&quot;syncingTo&quot; : &quot;xxxx.staging.member0.mongolayer.com:27017&quot;,&lt;br/&gt;
	&quot;members&quot; : [&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 3,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.member0.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 1,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;PRIMARY&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 1360494,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1320970851000,
				&quot;i&quot; : 19193
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2011-11-11T00:20:51Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2011-11-11T00:19:24Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 1&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 4,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.member1.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 2,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;SECONDARY&quot;,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1320970073000,
				&quot;i&quot; : 16237
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2011-11-11T00:07:53Z&quot;),&lt;br/&gt;
			&quot;self&quot; : true&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 5,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.arbiter.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 7,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;ARBITER&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 1360495,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 0,
				&quot;i&quot; : 0
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2011-11-11T00:19:24Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 0&lt;br/&gt;
		}&lt;br/&gt;
	],&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;mongostat on primary:&lt;br/&gt;
connected to: 127.0.0.1&lt;br/&gt;
insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn   set repl       time &lt;br/&gt;
     0      0      3      0       0       6       0  12.4g  12.9g  6.67g      1     98.1          0       0|0     0|2     1k     3k    54 xxxx    M   00:22:57 &lt;br/&gt;
     0      0      0      0       1       3       0  12.4g  12.9g  6.67g      2     91.7          0       0|0     0|2   324b     4m    54 xxxx    M   00:22:58 &lt;br/&gt;
     0      0      4      0       0       6       0  12.4g  12.9g  6.67g      1      102          0       0|0     0|2     1k     3k    54 xxxx    M   00:22:59 &lt;br/&gt;
     0      0      2      0       1       3       0  12.4g  12.9g  6.66g      0     96.3          0       0|0     0|2   565b     4m    54 xxxx    M   00:23:00 &lt;br/&gt;
     0      0      1      0       0       3       0  12.4g  12.9g  6.66g      0     99.7          0       0|0     0|2   277b     3k    54 xxxx    M   00:23:01 &lt;br/&gt;
     0      0      0      0       1       3       0  12.4g  12.9g  6.66g      0     93.5          0       0|1     1|2   277b     3k    54 xxxx    M   00:23:02 &lt;br/&gt;
     0      0      2      0       0       4       0  12.4g  12.9g  6.65g      2      0.5          0       0|2     1|3   400b     3k    54 xxxx    M   00:23:03 &lt;br/&gt;
     2      2      1      0       0       8       0  12.4g  12.9g  6.65g      0      5.4          0       0|1     1|2     1k     2k    54 xxxx    M   00:23:04 &lt;br/&gt;
     0      0      0      0       0       3       0  12.4g  12.9g  6.65g      0      6.9          0       0|1     1|2   277b     3k    54 xxxx    M   00:23:05 &lt;br/&gt;
     0      0      0      0       0       3       0  12.4g  12.9g  6.65g      0     86.3          0       0|0     0|2   324b     4m    54 xxxx    M   00:23:06&lt;/p&gt;

&lt;p&gt;mongostat on secondary:&lt;br/&gt;
connected to: 127.0.0.1&lt;br/&gt;
insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn   set repl       time &lt;br/&gt;
    *1     *0 *15565     *0       0     3|0       0  10.4g  10.9g  6.46g      0     97.8          0       0|0     0|0   277b     3k    31 xxxx  SEC   00:20:40 &lt;br/&gt;
    *0    100  *8376     *0       0     3|0       0  10.4g  10.9g  6.45g      7     49.4          0       0|0     0|0    13k    33k    31 xxxx  SEC   00:20:41 &lt;br/&gt;
    *2     *0 *14372     *0       0     3|0       0  10.4g  10.9g  6.46g      1       98          0       0|0     0|0   277b     3k    31 xxxx  SEC   00:20:42 &lt;br/&gt;
    *0     *0 *12544     *0       0     3|0       0  10.4g  10.9g  6.46g      0     82.6          0       0|0     0|0   277b     3k    31 xxxx  SEC   00:20:43 &lt;br/&gt;
    *0    100 *10449     *0       0     3|0       0  10.4g  10.9g  6.46g      7     61.5          0       0|0     0|0    13k    34k    31 xxxx  SEC   00:20:44 &lt;br/&gt;
    *0     *0 *14510     *0       0     3|0       0  10.4g  10.9g  6.46g      0     97.9          0       0|0     0|0   277b     3k    31 xxxx  SEC   00:20:45 &lt;br/&gt;
    *0     *0  *3186     *0       0     3|0       0  10.4g  10.9g  6.46g      0     22.1          0       0|0     0|0   277b     3k    31 xxxx  SEC   00:20:46 &lt;br/&gt;
    *0     45 *11096     *0       0     3|0       0  10.4g  10.9g  6.46g      4     70.1          0       0|0     0|0     6k    17k    31 xxxx  SEC   00:20:47 &lt;br/&gt;
    *0     42 *14068     *0       0     2|0       0  10.4g  10.9g  6.46g      2     91.2          0       0|0     0|0     5k    14k    31 xxxx  SEC   00:20:48 &lt;br/&gt;
    *0     38  *2986     *0       0     3|0       0  10.4g  10.9g  6.46g      1     14.4          0       0|0     0|0     5k    15k    31 xxxx  SEC   00:20:49&lt;/p&gt;

&lt;p&gt;Client says there was nothing going on at the time ... this one is their staging box. Looks like it is resyncing ... like perhaps the oplog rolls over but they claim they are not making that many changes and you can see from the primary that it doesn&apos;t do much. Can we inspect the oplog and look for large blasts of updates easily?&lt;/p&gt;

&lt;p&gt;While I was typing this out it got too stale:&lt;/p&gt;

&lt;p&gt;PRIMARY&amp;gt; rs.status()&lt;br/&gt;
{&lt;br/&gt;
	&quot;set&quot; : &quot;xxxx&quot;,&lt;br/&gt;
	&quot;date&quot; : ISODate(&quot;2011-11-11T00:41:20Z&quot;),&lt;br/&gt;
	&quot;myState&quot; : 1,&lt;br/&gt;
	&quot;syncingTo&quot; : &quot;xxxx.staging.member1.mongolayer.com:27017&quot;,&lt;br/&gt;
	&quot;members&quot; : [&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 3,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.member0.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 1,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;PRIMARY&quot;,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1320972080000,
				&quot;i&quot; : 5084
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2011-11-11T00:41:20Z&quot;),&lt;br/&gt;
			&quot;self&quot; : true&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 4,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.member1.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 3,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;RECOVERING&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 1362224,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 1320970605000,
				&quot;i&quot; : 1154
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;2011-11-11T00:16:45Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2011-11-11T00:41:19Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 0,&lt;br/&gt;
			&quot;errmsg&quot; : &quot;error RS102 too stale to catch up&quot;&lt;br/&gt;
		},&lt;br/&gt;
		{&lt;br/&gt;
			&quot;_id&quot; : 5,&lt;br/&gt;
			&quot;name&quot; : &quot;xxxx.staging.arbiter.mongolayer.com:27017&quot;,&lt;br/&gt;
			&quot;health&quot; : 1,&lt;br/&gt;
			&quot;state&quot; : 7,&lt;br/&gt;
			&quot;stateStr&quot; : &quot;ARBITER&quot;,&lt;br/&gt;
			&quot;uptime&quot; : 1362224,&lt;br/&gt;
			&quot;optime&quot; : &lt;/p&gt;
{
				&quot;t&quot; : 0,
				&quot;i&quot; : 0
			}
&lt;p&gt;,&lt;br/&gt;
			&quot;optimeDate&quot; : ISODate(&quot;1970-01-01T00:00:00Z&quot;),&lt;br/&gt;
			&quot;lastHeartbeat&quot; : ISODate(&quot;2011-11-11T00:41:20Z&quot;),&lt;br/&gt;
			&quot;pingMs&quot; : 0&lt;br/&gt;
		}&lt;br/&gt;
	],&lt;br/&gt;
	&quot;ok&quot; : 1&lt;br/&gt;
}&lt;/p&gt;

&lt;p&gt;So it looks like it might just be an issue of rolling the oplog ... even though updates look low.&lt;/p&gt;</comment>
                            <comment id="66024" author="eliot" created="Thu, 10 Nov 2011 21:49:12 +0000"  >&lt;p&gt;Does the secondary go into recovering or stay secondary throughout?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 10 Nov 2011 21:49:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 14 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 14 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>benwyrosdick</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrom0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrit0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23353</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsfy47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>