<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:04:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4086] OpenCursors spikes and needs primary reset to recover</title>
                <link>https://jira.mongodb.org/browse/SERVER-4086</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Something happens and our open cursors start climbing.  We alert internally when it hits a number much higher than normal (55).  To recover, we restart the primary.  Stopping the primary sometimes fails and requires kill -9.  We regularly lose data when we stop the primary (we&apos;re ok with some data loss, but the loss seems extreme - orders of magnitude more than shutting down a mysql master and moving to a replica).  Note that this happens several times per week (it can happen 5-10 times in a 24 hour period).  The last event started at ~12:50am PDT, 10/17/2011 (our alarm went off at ~2:00am and the master was restarted at ~2:20am).&lt;/p&gt;

&lt;p&gt;See mms:&lt;br/&gt;
clsol:PRIMARY&amp;gt; rs.conf()  &lt;br/&gt;
{&lt;br/&gt;
        &quot;_id&quot; : &quot;clsol&quot;,&lt;br/&gt;
        &quot;version&quot; : 1,&lt;br/&gt;
        &quot;members&quot; : [&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 0,
                        &quot;host&quot; : &quot;ec2-50-17-247-64.compute-1.amazonaws.com:27017&quot;
                }
&lt;p&gt;,&lt;br/&gt;
                &lt;/p&gt;
{
                        &quot;_id&quot; : 1,
                        &quot;host&quot; : &quot;ec2-50-17-247-65.compute-1.amazonaws.com:27017&quot;
                }
&lt;p&gt;,&lt;/p&gt;
                {
                        &quot;_id&quot; : 2,
                        &quot;host&quot; : &quot;ec2-50-17-247-66.compute-1.amazonaws.com:27017&quot;
                }
&lt;p&gt;        ]&lt;br/&gt;
}&lt;/p&gt;</description>
                <environment>Linix, 1.8.3, aws:m2.4xlarge, filesystem on tmpfs</environment>
        <key id="23713">SERVER-4086</key>
            <summary>OpenCursors spikes and needs primary reset to recover</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kristina">Kristina Chodorow</assignee>
                                    <reporter username="bpitman@netflix.com">Brent Pitman</reporter>
                        <labels>
                    </labels>
                <created>Mon, 17 Oct 2011 17:17:41 +0000</created>
                <updated>Mon, 24 Oct 2011 23:58:02 +0000</updated>
                            <resolved>Mon, 24 Oct 2011 23:57:51 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="62389" author="bpitman@netflix.com" created="Mon, 24 Oct 2011 23:52:51 +0000"  >&lt;p&gt;Please close this ticket and move discussion to Community Private SUPPORT-169.&lt;/p&gt;</comment>
                            <comment id="62329" author="kristina" created="Mon, 24 Oct 2011 20:35:53 +0000"  >&lt;p&gt;You can open a case in &lt;a href=&quot;https://jira.mongodb.org/browse/SUPPORT&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SUPPORT&lt;/a&gt; and it&apos;ll only be accessible to you and 10gen.  &lt;/p&gt;

&lt;p&gt;Have you tried running db.currentOp() when the number of cursors gets high?  If not, can you try that next time?&lt;/p&gt;</comment>
                            <comment id="62324" author="bpitman@netflix.com" created="Mon, 24 Oct 2011 20:00:34 +0000"  >&lt;p&gt;This appears to be a public viewable ticket, so we can&apos;t attach the log file.  It&apos;s 18MB (compressed).  How would yo like us to get it to you?&lt;/p&gt;</comment>
                            <comment id="62321" author="bpitman@netflix.com" created="Mon, 24 Oct 2011 19:47:48 +0000"  >&lt;p&gt;It happened again early this morning at nearly the same time as last week.  Attached is a graph showing week-over-week open cursors.  In bath cases, we &quot;recovered&quot; by restarting the primary.&lt;/p&gt;</comment>
                            <comment id="61131" author="spencer" created="Tue, 18 Oct 2011 15:03:32 +0000"  >&lt;p&gt;Can you attach your mongod logs?  They generally compress well from all the repeated text, so if you zip/tar them they should be quite small.&lt;/p&gt;</comment>
                            <comment id="61063" author="bpitman@netflix.com" created="Tue, 18 Oct 2011 05:10:25 +0000"  >&lt;p&gt;We use the perl driver.&lt;/p&gt;

&lt;p&gt;I&apos;m not sure how to characterize the queries.  The main table has 4 fields and ~20M entries.  Two of them (node,ds) are used for most queries.  There is an index and most queries have zero or one match, but can have as many as three.  We do two expensive queries to find all distinct nodes (~5000 of them, reuse the index) and all distinct slots (~50 of them, no index).  We also do queries for all filenames for a slot (~250,000) and all ds for a node (~10,000).  There don&apos;t appear to be any system resources being stressed.&lt;/p&gt;</comment>
                            <comment id="61038" author="eliot" created="Tue, 18 Oct 2011 02:27:29 +0000"  >&lt;p&gt;What kinds of queries are you doing?&lt;br/&gt;
What driver?&lt;/p&gt;</comment>
                            <comment id="60994" author="bpitman@netflix.com" created="Mon, 17 Oct 2011 21:42:59 +0000"  >&lt;p&gt;I don&apos;t know what&apos;s causing the high cursors.  We know that as it gets worse, we start timing out and retrying more (probably adding to the problem).  But we don&apos;t know what starts it.&lt;/p&gt;</comment>
                            <comment id="60919" author="bpitman@netflix.com" created="Mon, 17 Oct 2011 19:25:13 +0000"  >&lt;p&gt;Data loss depends on what else is going on.  Our max insert rate into mongodb seems to be ~200k per min (I&apos;d like to work with you to see if we can increase this by 1-2 orders of magnitude).  With a 10 sec delay, we&apos;re looking at 30k+ lost.&lt;/p&gt;

&lt;p&gt;I&apos;ll look into the w=2 option.  We&apos;re storing data related to metrics, so we&apos;re pretty sensitive to latency (if things are too slow, we can choose to either fall behind and drop data - we choose the latter).  I need to see what w=2 does to overall latency/thoughput.&lt;/p&gt;</comment>
                            <comment id="60914" author="eliot" created="Mon, 17 Oct 2011 19:12:51 +0000"  >&lt;p&gt;How much data loss are you seeing?&lt;/p&gt;

&lt;p&gt;One option to throttle is it to have the code doing the writes occasionally do a w=2.&lt;/p&gt;

&lt;p&gt;Do you know what&apos;s causing the high cursors?&lt;/p&gt;</comment>
                            <comment id="60912" author="bpitman@netflix.com" created="Mon, 17 Oct 2011 19:08:48 +0000"  >&lt;p&gt;Journaling is off by default, correct?&lt;/p&gt;

&lt;p&gt;mongod   20187     1  3 Oct13 ?        03:54:54 /usr/bin/mongod -f /etc/mongod.conf&lt;/p&gt;

&lt;p&gt;logpath = /logs/mongo/mongod.log&lt;br/&gt;
logappend=true&lt;br/&gt;
fork = true&lt;br/&gt;
dbpath = /dev/shm/mongod&lt;br/&gt;
replSet = clsol&lt;br/&gt;
rest = true&lt;br/&gt;
oplogSize = 4096&lt;/p&gt;


&lt;p&gt;The replication lag is variable.  Our app is generally busy during the first 1/2 of the min and much less so during the second half.  During the first half, it&apos;s common to see a lag of 5-10 secs.  It&apos;s usually 0-1 sec during the latter half.  However, when the number of cursors gets really high (1000&apos;s), we see the replication lag exceed 15 minutes.  Now that we alert when we reach 55 open cursors, though, we should never see replication get this far behind.&lt;/p&gt;</comment>
                            <comment id="60895" author="eliot" created="Mon, 17 Oct 2011 17:39:56 +0000"  >&lt;p&gt;Are you running with journalling?&lt;/p&gt;

&lt;p&gt;What&apos;s the normal secondary lag?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="13583" name="mongodb_cursors.png" size="31021" author="bpitman@netflix.com" created="Mon, 24 Oct 2011 19:47:48 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 17 Oct 2011 17:39:56 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 17 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 17 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bpitman@netflix.com</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kristina</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hronwv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hriqnr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22972</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0dpj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>