<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:07:22 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4927] Slaves stops replog sync if another slaves used fsyncLock</title>
                <link>https://jira.mongodb.org/browse/SERVER-4927</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;This is our Setup&lt;br/&gt;
2 Shards, each a ReplicaSet with 4 Nodes. 1 Node is dedicated for backups (priority:0,hidden:true)&lt;br/&gt;
If we start a backup we send the backup node the fsyncLock command and then start a rsync of the filesystem.&lt;br/&gt;
After we have finished the backup we send the fsyncUnLock command to the backup node.&lt;/p&gt;

&lt;p&gt;If we have a master switch (due to upgrade or failure) in the ReplicaSet we encounter the problem that some or all slaves stops oplog syncing when the backup node starts the backup. It is exactly the same moment as we start the fsyncLock command, since the replication lag is the same for the backup nodes and the slaves which also stops syncing. When the backup is finished the other slaves also starts syncing again. &lt;br/&gt;
db.currentOp() doesn&apos;t show the fsyncLock on the slaves, only on the backup node.&lt;br/&gt;
To get rid of this problem we have to start the non backup slave. After this restart the slave runs well and never stop syncing again together with the backup node.&lt;/p&gt;

&lt;p&gt;This is the second time we&apos;ve encoutered this problem. Since this is our production environment we don&apos;t want to force a master switch if not needed.&lt;/p&gt;

&lt;p&gt;It seems that the cause of this problem is the master switch in the replicaset.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Steffen&lt;/p&gt;</description>
                <environment>Linux 2.6.32-38-server, Ubuntu 10.04, MongoDB 2.0.1, Replicaset with 4 Nodes, NUMA, 2x XEON E5620 , 24 GB RAM </environment>
        <key id="30344">SERVER-4927</key>
            <summary>Slaves stops replog sync if another slaves used fsyncLock</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="backlog-server-repl">Backlog - Replication Team</assignee>
                                    <reporter username="steffen">Steffen</reporter>
                        <labels>
                            <label>sync</label>
                    </labels>
                <created>Fri, 10 Feb 2012 13:52:31 +0000</created>
                <updated>Tue, 6 Dec 2022 05:36:33 +0000</updated>
                            <resolved>Thu, 22 Feb 2018 20:45:02 +0000</resolved>
                                    <version>2.0.2</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="1813442" author="greg.mckeon" created="Thu, 22 Feb 2018 20:45:02 +0000"  >&lt;p&gt;We believe this has gone away - if this is still an issue, please feel free to file a new ticket.&lt;/p&gt;</comment>
                            <comment id="284780" author="milkie" created="Fri, 8 Mar 2013 22:32:44 +0000"  >&lt;p&gt;This should now behave somewhat better, in that the secondaries might pause for a bit but after 30 seconds should switch sync sources away from the locked node and catch up.&lt;/p&gt;</comment>
                            <comment id="96599" author="kristina" created="Wed, 7 Mar 2012 22:37:20 +0000"  >&lt;p&gt;The stuck secondaries were probably syncing from the fsync+locked secondary.  The secondaries should recalculate who to sync from periodically.&lt;/p&gt;</comment>
                            <comment id="96334" author="steffen" created="Wed, 7 Mar 2012 13:36:45 +0000"  >&lt;p&gt;So far the problem did not happen again. We are in the process of migrating all the host and also upgrade to 2.0.3.&lt;br/&gt;
For the upgrades and the migration there is going to be a master switch. We will monitor if the problem reoccurs than.&lt;/p&gt;</comment>
                            <comment id="96331" author="steffen" created="Wed, 7 Mar 2012 13:34:27 +0000"  >&lt;p&gt;No, we don&apos;t use authentication.&lt;/p&gt;</comment>
                            <comment id="96329" author="eliot" created="Wed, 7 Mar 2012 13:33:11 +0000"  >&lt;p&gt;Are you running with authentication?&lt;/p&gt;</comment>
                            <comment id="87266" author="steffen" created="Fri, 10 Feb 2012 14:49:43 +0000"  >&lt;p&gt;On the backup node we see the fsyncLock process. I don&apos;t have the db.currentOp() form the last time.&lt;br/&gt;
Log from backup node:&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 172.20.4.219:21561 #403654&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn403654&amp;#93;&lt;/span&gt; CMD fsync:  sync:1 lock:1&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn403654&amp;#93;&lt;/span&gt; removeJournalFiles&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;fsyncjob&amp;#93;&lt;/span&gt; db is now locked for snapshotting, no writes allowed. db.fsyncUnlock() to unlock&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;fsyncjob&amp;#93;&lt;/span&gt;     For more info see &lt;a href=&quot;http://www.mongodb.org/display/DOCS/fsync+Command&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://www.mongodb.org/display/DOCS/fsync+Command&lt;/a&gt;&lt;br/&gt;
Fri Feb 10 05:59:35 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn403654&amp;#93;&lt;/span&gt; command admin.$cmd command: &lt;/p&gt;
{ fsync: 1.0, lock: true }
&lt;p&gt; ntoreturn:1 reslen:168 246ms&lt;/p&gt;

&lt;p&gt;We don&apos;t see this log entry in the slave logs.&lt;/p&gt;

&lt;p&gt;This time we had 1 slave which was 1 second behind the primary. The second slave and the backup node had the same lag, which increased over time.&lt;br/&gt;
We get this values using the nagios check for mongodb (&lt;a href=&quot;http://tag1consulting.com/blog/mongodb-nagios-check&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://tag1consulting.com/blog/mongodb-nagios-check&lt;/a&gt;).&lt;br/&gt;
We also have MMS agent running. This should also show the replication lag?&lt;/p&gt;

&lt;p&gt;If my guess is correct, the next time this happens will be after a master switch. &lt;br/&gt;
Possible is also that the other slave which wasn&apos;t stale hangs the next time the backup runs, because I haven&apos;t restarted it yet.&lt;/p&gt;</comment>
                            <comment id="87250" author="scotthernandez" created="Fri, 10 Feb 2012 14:22:47 +0000"  >&lt;p&gt;Is the primary farther away from the secondaries than the backup replica? Can you run db.currentOp() on the backup replica the next time this happens? Also, running db.getReplicationInfo() on each of the other replicas would be helpful.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="32219">SERVER-5208</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25128"><![CDATA[Replication]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 10 Feb 2012 14:22:47 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 50 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-5208'>SERVER-5208</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 50 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-repl</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>milkie@mongodb.com</customfieldvalue>
            <customfieldvalue>greg.mckeon@mongodb.com</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
            <customfieldvalue>steffen</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrodtr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrft13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4882</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrlgq7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>