<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:06:41 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-4684] Severe server slowdown</title>
                <link>https://jira.mongodb.org/browse/SERVER-4684</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;One of our servers started expecting severe slowdowns (working 10-100x slower) around 17:05 (time for reference only). It continued working, but very slowly, leading to other replica set deciding it&apos;s dead around 17:29.&lt;br/&gt;
I explored the log, and aside the &quot;long query&quot; warnings (queries that took 100-200ms started taking around 8000ms) there was this one:&lt;br/&gt;
Sun Jan 15 17:10:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.2.1.200:36130 #169664&lt;br/&gt;
...&lt;br/&gt;
Sun Jan 15 17:10:24 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn169664&amp;#93;&lt;/span&gt;   Assertion failure cc-&amp;gt;_pinValue &amp;lt; 100 db/clientcursor.h 309&lt;br/&gt;
0x57a926 0x5857db 0x7982e9 0x882846 0x889e99 0xaa00c6 0x635bb7 0x7fa26256c9ca 0x7fa261b1b70d&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0x96) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x57a926&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0xfb) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x5857db&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod() &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7982e9&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo19receivedKillCursorsERNS_7MessageE+0xb6) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x882846&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x359) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x889e99&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) &lt;span class=&quot;error&quot;&gt;&amp;#91;0xaa00c6&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x635bb7&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libpthread.so.0(+0x69ca) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fa26256c9ca&amp;#93;&lt;/span&gt;&lt;br/&gt;
 /lib/libc.so.6(clone+0x6d) &lt;span class=&quot;error&quot;&gt;&amp;#91;0x7fa261b1b70d&amp;#93;&lt;/span&gt;&lt;br/&gt;
...&lt;br/&gt;
Sun Jan 15 17:10:23 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 10.2.1.200:36130 #169664&lt;/p&gt;

&lt;p&gt;Right around the problem start we saw these (there was NO mentioning of any &quot;killcursors&quot; before in the log):&lt;br/&gt;
Sun Jan 15 17:05:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn169305&amp;#93;&lt;/span&gt; killcursors  64ms&lt;br/&gt;
Sun Jan 15 17:05:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn169296&amp;#93;&lt;/span&gt; killcursors  64ms&lt;br/&gt;
Sun Jan 15 17:05:45 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn169309&amp;#93;&lt;/span&gt; killcursors  60ms&lt;/p&gt;

&lt;p&gt;which became a lot worse later:&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171423&amp;#93;&lt;/span&gt; killcursors  2396ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171404&amp;#93;&lt;/span&gt; killcursors  2396ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171396&amp;#93;&lt;/span&gt; killcursors  2396ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171428&amp;#93;&lt;/span&gt; killcursors  2392ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171398&amp;#93;&lt;/span&gt; killcursors  2392ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171459&amp;#93;&lt;/span&gt; killcursors  2392ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171436&amp;#93;&lt;/span&gt; killcursors  2392ms&lt;br/&gt;
Sun Jan 15 17:28:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn171334&amp;#93;&lt;/span&gt; killcursors  2492ms&lt;/p&gt;

&lt;p&gt;Maybe it&apos;s our application mismanaging cursors?&lt;/p&gt;


&lt;p&gt;At first I decided that there was some kind of a hard drive failure, leading to this, but just restarting the server fixed the problem completely, so I don&apos;t think it&apos;s a hardware problem.&lt;br/&gt;
The database is 99% read-only. the load is medium-light.&lt;/p&gt;

&lt;p&gt;My guess is either some internal structures got damaged, or there is some problem with PHP driver or our pattern of it&apos;s usage.&lt;/p&gt;

&lt;p&gt;I checked dmesg, kern.log, etc. &amp;#8211; no signs of anything going wrong with hardware and/or kernel stuff.&lt;br/&gt;
Also, I checked historical atop data, between 16:30-17:00 (before the problem) it did 48120KiB reads and 5936KiB writes using around 7% CPU, and between 17:00-17:30 (the problematic interval) it did 175.2MiB reads and 142.2MiB writes using around 10% CPU (3 times more reads and 30 times more writes) &amp;#8211; so it looks like it did a lot of unexpected I/O that our application didn&apos;t request (TCP I/O grew up around 30% max). Swap space kept mostly unused, file cache was the same.&lt;/p&gt;

&lt;p&gt;This problem recurs without any visible causes, please advise.&lt;/p&gt;</description>
                <environment></environment>
        <key id="28346">SERVER-4684</key>
            <summary>Severe server slowdown</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="onyxmaster">Aristarkh Zagorodnikov</reporter>
                        <labels>
                            <label>pv1</label>
                    </labels>
                <created>Sun, 15 Jan 2012 14:14:23 +0000</created>
                <updated>Sun, 15 Jan 2012 16:28:07 +0000</updated>
                            <resolved>Sun, 15 Jan 2012 16:28:07 +0000</resolved>
                                    <version>2.0.2</version>
                                                                        <votes>0</votes>
                                    <watches>0</watches>
                                                                                                                <comments>
                            <comment id="79956" author="eliot" created="Sun, 15 Jan 2012 16:28:07 +0000"  >&lt;p&gt;there are other cases related to the assert - see the label, so going to close this&lt;/p&gt;</comment>
                            <comment id="79952" author="onyxmaster" created="Sun, 15 Jan 2012 15:04:42 +0000"  >&lt;p&gt;Additional details: we had this problem all along, but VM guys allowed burst IOPS to be 8x higher for short periods of time. When load came, the burst gone and we got stuck with consumer-HDD-grade speeds for all our LVMs.&lt;/p&gt;</comment>
                            <comment id="79951" author="onyxmaster" created="Sun, 15 Jan 2012 15:02:53 +0000"  >&lt;p&gt;Sorry for the false alert, working with our VM provider people proved that there is a hidden binding of IOPS slots to memory slots, so having low memory on machines (we set 2Gb RAM) leaded to IOPS being limited to 300 IOPS per device that thrashed everything. Sorry again for this, it appears you can close the case (although that assert might still need some attention).&lt;/p&gt;</comment>
                            <comment id="79949" author="onyxmaster" created="Sun, 15 Jan 2012 14:48:29 +0000"  >&lt;p&gt;It &lt;em&gt;appears&lt;/em&gt; that there was a problem with limited memory on a VM. Increasing amount of RAM and IOPS slots (we have pay-as-you-use virtualization billing) got rid of the problems temporarily, I will report progress later.&lt;/p&gt;</comment>
                            <comment id="79948" author="onyxmaster" created="Sun, 15 Jan 2012 14:39:30 +0000"  >&lt;p&gt;php driver is 1.2.6&lt;/p&gt;</comment>
                            <comment id="79947" author="eliot" created="Sun, 15 Jan 2012 14:20:38 +0000"  >&lt;p&gt;What version of the php driver?&lt;/p&gt;</comment>
                            <comment id="79946" author="onyxmaster" created="Sun, 15 Jan 2012 14:19:00 +0000"  >&lt;p&gt;Query execution skyrockets after the problem triggers, check this, the longer, the worse:&lt;br/&gt;
Sun Jan 15 18:16:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn914&amp;#93;&lt;/span&gt; query drive.entries2 nscanned:7723 nreturned:101 reslen:95203 886150ms&lt;br/&gt;
Sun Jan 15 18:16:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1063&amp;#93;&lt;/span&gt; query drive.entries2 nscanned:7723 nreturned:101 reslen:95203 808726ms&lt;br/&gt;
Sun Jan 15 18:16:02 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn1436&amp;#93;&lt;/span&gt; query drive.entries2 nscanned:7723 nreturned:101 reslen:95203 648782ms&lt;/p&gt;</comment>
                            <comment id="79945" author="onyxmaster" created="Sun, 15 Jan 2012 14:17:30 +0000"  >&lt;p&gt;I would like to comment that this runs on Xen virtual environment, using 2.6.32.36 kernel.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Sun, 15 Jan 2012 14:20:38 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 5 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 5 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>onyxmaster</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrogvb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hritnb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23456</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht0dg7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>