<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:47:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-18512] Performance drop after migrating from 2.4 to 2.6</title>
                <link>https://jira.mongodb.org/browse/SERVER-18512</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Immediately after migrating a mongodb replica set from 2.4.5 to 2.6.9 we observed an increased lock % on all members, even on secondaries that were only synchronizing without serving queries.&lt;br/&gt;
Our replica set is composed by 4 nodes -&amp;gt; 1 primary, 2 slaves and an hidden one. We do not use sharding. We rolled back all nodes except a slave in order to troubleshoot and diagnose the issue.&lt;br/&gt;
We attached iostats output and mongo mms (please consider interval 09-11 AM) graphs collected at the same time on the two slave nodes with different mongo versions.&lt;br/&gt;
As you can see the node with mongo 2.6 shows increased lock % and background flush avg. Also the amount of data written to the recovery log and database datafiles per second are much more bigger on mongo 2.6.&lt;/p&gt;</description>
                <environment></environment>
        <key id="204819">SERVER-18512</key>
            <summary>Performance drop after migrating from 2.4 to 2.6</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="5">Cannot Reproduce</resolution>
                                        <assignee username="sam.kleinman">Sam Kleinman</assignee>
                                    <reporter username="mrizzo">Marzio Rizzo</reporter>
                        <labels>
                    </labels>
                <created>Mon, 18 May 2015 13:19:27 +0000</created>
                <updated>Wed, 28 Oct 2015 18:53:34 +0000</updated>
                            <resolved>Wed, 28 Oct 2015 18:53:34 +0000</resolved>
                                    <version>2.6.8</version>
                    <version>2.6.9</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1073187" author="ramon.fernandez" created="Wed, 28 Oct 2015 18:53:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mrizzo&quot; class=&quot;user-hover&quot; rel=&quot;mrizzo&quot;&gt;mrizzo&lt;/a&gt;, apologies for the radio silence. We have been unable to reproduce this behavior on our end after trying in different environments, and we&apos;re not sure what may be different about your environment that causes the performance drop you report, so unfortunately there&apos;s not much more investigation we can do on our end.&lt;/p&gt;

&lt;p&gt;That being said, MongoDB 3.0 includes numerous improvements over 2.6, including collection-level locking for MMAPv1, so I&apos;d encourage you to try out 3.0.7 (the latest stable release at the time of this writing) and see if it performs better than 2.4/2.6 for your use case.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="940156" author="mrizzo" created="Mon, 15 Jun 2015 10:41:07 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ramon.fernandez&quot; class=&quot;user-hover&quot; rel=&quot;ramon.fernandez&quot;&gt;ramon.fernandez&lt;/a&gt;, &lt;br/&gt;
for easily reproduce this behaviour you can proceed this way:&lt;/p&gt;

&lt;p&gt;1. Install mongo-2.4.5 on all 3 nodes&lt;br/&gt;
2. Run &quot;500k_populator.js&quot; on the primary in order to populate db&lt;br/&gt;
3. Run &quot;mongo test 500k_updates_benchRun.js&quot; for load generation on the primary&lt;br/&gt;
4. Run mongostat on every node and see the results (in particular the number of updates and the locked db %)&lt;br/&gt;
5. Upgrade to mongo-2.6.9 all 3 nodes&lt;br/&gt;
6. Run &quot;mongo test 500k_updates_benchRun.js&quot; for load generation on the primary&lt;br/&gt;
7. Run mongostat on every node and see the results (in particular the number of updates and the locked db %)&lt;/p&gt;

&lt;p&gt;We attached the mongo_replicaset_test_results.xlsx report with our results.&lt;br/&gt;
As you can see the nodes with mongo 2.4.5 have more updates/sec with less locked db % than the nodes with mongo 2.6.9.&lt;br/&gt;
It is normal this behaviour?&lt;/p&gt;</comment>
                            <comment id="932730" author="mrizzo" created="Fri, 5 Jun 2015 16:03:34 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ramon.fernandez&quot; class=&quot;user-hover&quot; rel=&quot;ramon.fernandez&quot;&gt;ramon.fernandez&lt;/a&gt;, could you please provide us the mongodb configuration that you&apos;re using during your tests? &lt;br/&gt;
It could be very usefull for us to understand how our and your results can be so different. &lt;br/&gt;
We attached our mongodb.conf and our replicaset configuration for comparison.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Marzio&lt;/p&gt;</comment>
                            <comment id="927729" author="ramon.fernandez" created="Fri, 29 May 2015 19:15:10 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mrizzo&quot; class=&quot;user-hover&quot; rel=&quot;mrizzo&quot;&gt;mrizzo&lt;/a&gt;, I have not been able to reproduce this behavior on my end, and in fact I&apos;m observing the opposite effect: in 2.4.5 the lock% hovers around 70% and I get around 75K updates/second, while in 2.6.9 the lock% hovers around 50% and I get around 155K updates/second.&lt;/p&gt;

&lt;p&gt;I looked at the logs you sent and they just contain startup data, so not much information there that can help us understand what your replica set is doing. I also looked at your MMS graphs and I don&apos;t see anything strange happening from 9-11am (except for the lock% difference). While I do see the data from &lt;tt&gt;mongostat&lt;/tt&gt; that you obtained I haven&apos;t been able to observe the same behavior.&lt;/p&gt;

&lt;p&gt;Is there other information you may be able to provide that can be of help here? Are you using any other options for your {{mongod}}s like powerOf2Sizes, noprealloc or smallfiles by any chance? Anything else you can think of?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="925715" author="ramon.fernandez" created="Thu, 28 May 2015 11:59:52 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=mrizzo&quot; class=&quot;user-hover&quot; rel=&quot;mrizzo&quot;&gt;mrizzo&lt;/a&gt;, we haven&apos;t been able to reproduce this behavior as of yet but we continue to investigate. We&apos;ll post updates to this ticket when we have more information.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="925626" author="mrizzo" created="Thu, 28 May 2015 07:51:57 +0000"  >&lt;p&gt;Hi guys, do you need any further information/material? Did you find useful the attached inputs? Thanks, best regards&lt;/p&gt;</comment>
                            <comment id="921674" author="mrizzo" created="Fri, 22 May 2015 15:08:16 +0000"  >&lt;p&gt;We forgot to attach mongostat output in which you can see the lock % of all mongod during the load test (ended at 16:53:07).&lt;/p&gt;</comment>
                            <comment id="921669" author="mrizzo" created="Fri, 22 May 2015 15:06:10 +0000"  >&lt;p&gt;We attached the requested logs. Let us know if you need more details.&lt;br/&gt;
Thanks for your support.&lt;/p&gt;</comment>
                            <comment id="920238" author="ramon.fernandez" created="Thu, 21 May 2015 12:35:10 +0000"  >&lt;p&gt;Can you please upload the log files for the primary and the 2.6.9 secondary from startup until you run your load test? The &quot;from startup&quot; part is to see how are your &lt;tt&gt;mongod&lt;/tt&gt; configured, but if the logs are too large you can upload the part corresponding to the load test and the configuration files used for each &lt;tt&gt;mongod&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="917871" author="mrizzo" created="Tue, 19 May 2015 14:05:42 +0000"  >&lt;p&gt;OS is Centos 6.5 64 bit with kernel 2.6.32-431.el6.x86_64.&lt;br/&gt;
We easily reproduced the issue on a test environment componsed by 3 physical nodes (a master and two slaves, one of the slaves runs mongodb 2.4.5, all other nodes run mongodb 2.6.9). Follow this steps:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Run &quot;mongo test 500k_populator.js&quot; on the primary in order to populate db&lt;/li&gt;
	&lt;li&gt;Run &quot;mongo test 500k_updates_benchRun.js&quot; for load generation on the primary&lt;/li&gt;
	&lt;li&gt;Run mongostat on both slaves and see the &quot;locked db&quot; column. Values on 2.6.9 slave are at least double&lt;/li&gt;
	&lt;li&gt;Other comparison can be made with running commands &quot;iostat -xmt 1&quot; or &quot;sar -b 1 100&quot;&lt;/li&gt;
&lt;/ol&gt;
</comment>
                    </comments>
                    <attachments>
                            <attachment id="74862" name="500k_populator.js" size="220" author="mrizzo" created="Tue, 19 May 2015 14:05:42 +0000"/>
                            <attachment id="74863" name="500k_updates_benchRun.js" size="428" author="mrizzo" created="Tue, 19 May 2015 14:05:42 +0000"/>
                            <attachment id="74684" name="iostats.xlsx" size="18235" author="mrizzo" created="Mon, 18 May 2015 13:19:27 +0000"/>
                            <attachment id="75396" name="mongo-2.4.5-secondary.tar.gz" size="1239" author="mrizzo" created="Fri, 22 May 2015 15:06:10 +0000"/>
                            <attachment id="75397" name="mongo-2.6.9-primary.tar.gz" size="5563" author="mrizzo" created="Fri, 22 May 2015 15:06:10 +0000"/>
                            <attachment id="75398" name="mongo-2.6.9-secondary.tar.gz" size="1378" author="mrizzo" created="Fri, 22 May 2015 15:06:10 +0000"/>
                            <attachment id="74685" name="mongo24.png" size="304005" author="mrizzo" created="Mon, 18 May 2015 13:19:27 +0000"/>
                            <attachment id="74686" name="mongo26.png" size="277651" author="mrizzo" created="Mon, 18 May 2015 13:19:27 +0000"/>
                            <attachment id="77887" name="mongo_replicaset_test_results.xlsx" size="9335" author="mrizzo" created="Mon, 15 Jun 2015 10:41:07 +0000"/>
                            <attachment id="76883" name="mongod.conf" size="157" author="mrizzo" created="Fri, 5 Jun 2015 16:03:34 +0000"/>
                            <attachment id="75399" name="mongostat.png" size="313988" author="mrizzo" created="Fri, 22 May 2015 15:08:16 +0000"/>
                            <attachment id="76882" name="replicaset-configure.js" size="458" author="mrizzo" created="Fri, 5 Jun 2015 16:03:34 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 18 May 2015 14:58:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 16 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 16 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>mrizzo</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>sam.kleinman</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl5rj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hs9fxz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsg10f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>