<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:02:14 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3157] Replicaset becomes inaccessable and instable after mapreduce job</title>
                <link>https://jira.mongodb.org/browse/SERVER-3157</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;After we run a mapreduce job which updates thousends of records the primary mongodb server becomes inaccessable. It was not able to connect via PHP webnode or local mongo shell. In a short time the server reached his connection limit (in normal operation we have around 10/s; after the mapreduce job they step up to &amp;gt; 13000; the PHP webnodes use non-persistent conncetions; see lx03_mongostat_cutted.txt). the 13000 connection where full established but idle (see attachment overview_mongod_after_midnight.html). &lt;/p&gt;

&lt;p&gt;Our first action was to shutdown the php webserver nodes. connections jumps back to 10 and the system becomes accessable again.&lt;/p&gt;

&lt;p&gt;Second action was to shutdown the secondary and start the map reduce job again. everything run smooths seeming without probems. During the mapreduce job used ram increased steadily (see munin graphs). When the job was finished we start the secondary again. From here everything works as expected running the operations from oplog. After a short sleep we saw in the morning there was a connection jump again to 1000. So I decieded to stop and start the current primary and let the secondary take over to get a clean state again.&lt;/p&gt;

&lt;p&gt;The attachments containing mongostats, munin graphs, mongodb.logs and the home view from mongodb internal webserver. The munin graphs contains some leaks where the primary was inaccessable to gather data. (server lx03 is primary, lx04 is secondary)&lt;/p&gt;

&lt;p&gt;In the past we had the situation once or twice per month probably from a cron job starting another mapreduce operation. But until yesterday we couldn&apos;t track it down.&lt;/p&gt;</description>
                <environment>Replicat Set with 2 Nodes (24 GB RAM) + 2 Arbiter all openSuse 11.3 Kernel 2.6.34 64bit</environment>
        <key id="17594">SERVER-3157</key>
            <summary>Replicaset becomes inaccessable and instable after mapreduce job</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="-1">Unassigned</assignee>
                                    <reporter username="espe">thomas</reporter>
                        <labels>
                            <label>connection</label>
                            <label>mapreduce,</label>
                            <label>php</label>
                            <label>replication</label>
                    </labels>
                <created>Fri, 27 May 2011 11:56:04 +0000</created>
                <updated>Fri, 30 Mar 2012 14:37:13 +0000</updated>
                            <resolved>Sat, 4 Jun 2011 04:57:15 +0000</resolved>
                                    <version>1.8.1</version>
                                                                        <votes>1</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="35952" author="espe" created="Fri, 3 Jun 2011 15:15:04 +0000"  >&lt;p&gt;Currently we are unable to use another server for redis. It wouldn&apos;t be matter if only performance goes down for a moment. A bigger problem are the leaked connections (they are established but not used anymore - which could also be a php driver problem) and not being able to connect to the server anymore during the block. But I know it&apos;s a special case with our setup and hard to reproduce without the traffic and load of a production enviroment. At the moment it seems our stripped down solution is working. I still have to observe it and will inform you if the problem occurs again.&lt;/p&gt;</comment>
                            <comment id="35926" author="eliot" created="Fri, 3 Jun 2011 11:36:24 +0000"  >&lt;p&gt;Its possible that if something else on the box starts eating resources, mongo performance can spiral down as more connections pile on, etc...&lt;br/&gt;
Is it possible to split redis and mongo?&lt;/p&gt;</comment>
                            <comment id="35925" author="espe" created="Fri, 3 Jun 2011 11:29:58 +0000"  >&lt;p&gt;Yesterday I found out the problems could be related to a redis instance running on the same server. Writing the redis dump file needs around 15 sec. In this time the cpu usage of mongodb increase rapidly and new connections to mongodb begin to leak or failed. After redis has finished his dump, the mongodb logfile suddenly shows a bunch of operations needing more then 100ms to complete. When we shutdown the redis instance mongodb runs fine.&lt;/p&gt;

&lt;p&gt;The server has one partition only with a hardware RAID10 on SAS disks. As there is enough memory (24G) we thought it&apos;s no problem to run a redis instance beside mongodb. Also the disk and system related munin graphs shows no obvious problems in this direction, I think.&lt;/p&gt;

&lt;p&gt;Right now we changed the application to reduce the amount of connections per second made with mongodb - no problems occur since the change despite of running mongodb and redis in parallel. But unfortunately we cannot use it in the haviness we want.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="11952" name="logs_and_mongostat_complete.zip" size="511470" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                            <attachment id="11950" name="lx03_mongostat_cutted.txt" size="50943" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                            <attachment id="11951" name="munin_graphs.png" size="1698995" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                            <attachment id="11953" name="overview_mongod_after_midnight.html" size="130080" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                            <attachment id="11954" name="overview_mongod_normal.html" size="15488" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                            <attachment id="11955" name="overview_mongod_today_morning.html" size="170423" author="espe" created="Fri, 27 May 2011 11:56:04 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 3 Jun 2011 11:36:24 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 37 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ian@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 37 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>espe</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hroz33:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrg3lj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7485</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrj0af:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>