<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:14:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-27153] Mongodump on secondaries causes performance issue with w:majority</title>
                <link>https://jira.mongodb.org/browse/SERVER-27153</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;We are experiencing performance issues on the master when mongodump backs up on secondaries.&lt;/p&gt;

&lt;p&gt;Out applications all perform read / writes  from the master. The slaves are only used for replica set and backups.&lt;/p&gt;

&lt;p&gt;Our backups are performed at 8am / 8pm every day and can take a considerable amount of time to run on the slaves. &lt;/p&gt;

&lt;p&gt;Backups are run using the mongodump command as follows for each DB on the cluster.&lt;br/&gt;
&lt;tt&gt;&apos;/usr/bin/mongodump --host=localhost --gzip --db=210596-game-live --excludeCollection=requestsByUserState --out=/mnt/backup --quiet&apos;&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;Stopping the mongodump process from running on the secondaries resolves our masters performance issues.&lt;/p&gt;

&lt;p&gt;So far we have seen this issue on two of our mongo clusters. &lt;/p&gt;

&lt;p&gt;Please see details for one of the clusters we have seen this issue with.&lt;/p&gt;

&lt;p&gt;We usually run a larger master to the slaves to reduce cost as follows.&lt;/p&gt;

&lt;p&gt;gsp-aeu001-mo04 (M) r3.2xlarge &lt;br/&gt;
gsp-aeu001-mo05 (S) r3.xlarge &lt;br/&gt;
gsp-aeu001-mo06 (S) r3.xlarge&lt;/p&gt;

&lt;p&gt;At the high level we can see at 8am/pm the number of commands/querys/updates etc drop off for a period of time whilst the backups run. This then causes the java applications connecting to the master to experience queuing threads.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;144613_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/144613/144613_Master+1+day+commands+graph+.png&quot; title=&quot;Master 1 day commands graph .png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;144613&quot; file-preview-title=&quot;Master 1 day commands graph .png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/144613/_thumb_144613.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;At the same time as the commands fluctuate the performance of all three servers look as follows.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;144612_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/144612/144612_Cluster+cpu.png&quot; title=&quot;Cluster cpu.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;144612&quot; file-preview-title=&quot;Cluster cpu.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/144612/_thumb_144612.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;It appears that the performance of the secondaries backing up causes a degradation in performance on the master.&lt;/p&gt;

&lt;p&gt;We have gathered the &quot;/var/lib/mongodb/diagnostic.data/&quot; data for each of the servers. Do you have a none public place for us to upload this data please?&lt;/p&gt;

&lt;p&gt;Increasing the size of the Mongo secondaries to match the master has improved the performance, however we would like to understand why backups on secondaries cause performance issues on the master.&lt;/p&gt;

&lt;p&gt;Kind regards&lt;br/&gt;
Luke&lt;/p&gt;






</description>
                <environment>Production</environment>
        <key id="333604">SERVER-27153</key>
            <summary>Mongodump on secondaries causes performance issue with w:majority</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="lukemorfitt">Luke Morfitt</reporter>
                        <labels>
                    </labels>
                <created>Tue, 22 Nov 2016 13:31:52 +0000</created>
                <updated>Tue, 21 Feb 2017 21:48:35 +0000</updated>
                            <resolved>Tue, 21 Feb 2017 21:48:35 +0000</resolved>
                                    <version>3.2.10</version>
                                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1505498" author="thomas.schubert" created="Tue, 21 Feb 2017 21:48:35 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lukemorfitt&quot; class=&quot;user-hover&quot; rel=&quot;lukemorfitt&quot;&gt;lukemorfitt&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;m closing this ticket in favor of our investigation on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27753&quot; title=&quot;wiredTiger cache 95% performance issues&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27753&quot;&gt;&lt;del&gt;SERVER-27753&lt;/del&gt;&lt;/a&gt; as I believe that ticket addresses the same root cause.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1450018" author="lukemorfitt" created="Tue, 6 Dec 2016 09:17:01 +0000"  >&lt;p&gt;Correct. &lt;/p&gt;

&lt;p&gt;We split the number of databases we are hosting across the secondaries available and perform backups at 8am / 8pm. &lt;/p&gt;

&lt;p&gt;We aim to get a backup of every database every 12 hours. In order to do that we need to backup on both secondaries at the same time as our backups are rather time consuming. They would not complete on a single secondary within a 12 hour period.&lt;/p&gt;

&lt;p&gt;Whilst this may not be the usual use case this does not explain the global level locking.&lt;/p&gt;

&lt;p&gt;I would appreciate understanding the global level locking during the cache eviction process.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Luke&lt;/p&gt;

</comment>
                            <comment id="1449738" author="asya" created="Mon, 5 Dec 2016 22:05:45 +0000"  >&lt;p&gt;You say  &quot;both of the secondaries are performing backups&quot; - more common would be to use one of them for backups only and let the other one be the &quot;hot standby&quot; for the primary (and be the one available to acknowledge w:majority writes.&lt;/p&gt;</comment>
                            <comment id="1449410" author="lukemorfitt" created="Mon, 5 Dec 2016 17:33:20 +0000"  >&lt;p&gt;Hello,&lt;/p&gt;

&lt;p&gt;The current theory I have is as follows.&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Our application uses &quot;majority write concern&quot; requiring 2 of the 3 nodes to confirm a write has taken place.&lt;/li&gt;
	&lt;li&gt;During our backup window both of the secondaries are performing backups which is causing global locks during high wiriedTiger cache utilization 85%+&lt;/li&gt;
	&lt;li&gt;This then slows down the  &quot;majority write concern&quot; completing in a timely fashion and impacts the performance of our application.&lt;/li&gt;
&lt;/ol&gt;



&lt;p&gt;Increasing the size of the secondaries(providing more cache space) does not fix the issue as they still hit more than 85% cache usage and start to lock. The impact is reduced, however still apparent.&lt;/p&gt;

&lt;p&gt;Any thoughts on this matter would be greatly appreciated.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Luke&lt;/p&gt;</comment>
                            <comment id="1449056" author="lukemorfitt" created="Mon, 5 Dec 2016 14:07:09 +0000"  >&lt;p&gt;I&apos;ve noticed the following whilst trying to diagnose this issue.&lt;/p&gt;

&lt;p&gt;Upon running a mongodump with the following command we see locks appear on the secondary.&lt;/p&gt;

&lt;p&gt;&quot;/usr/bin/mongodump --host=localhost --gzip --db=DB_NAME --excludeCollection=requestsByUserState --out=./DB_NAME-vv&quot;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;145322_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/145322/145322_Secondary+running+mongodump.png&quot; title=&quot;Secondary running mongodump.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;145322&quot; file-preview-title=&quot;Secondary running mongodump.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/145322/_thumb_145322.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;At the same time the replication Commands on the master reduce. &lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;145323_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/145323/145323_master+commands.png&quot; title=&quot;master commands.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;145323&quot; file-preview-title=&quot;master commands.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/145323/_thumb_145323.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I would like to understand the cause of the locking taking place. Is it caused by the loading of data into the wiredTiger cache (as the cache gets to 95% and then loads more from disk).  Or is it caused by a requirement to lock the collection during a dump?&lt;/p&gt;

&lt;p&gt;Thanks&lt;br/&gt;
Luke&lt;/p&gt;


</comment>
                            <comment id="1441101" author="lukemorfitt" created="Wed, 23 Nov 2016 14:30:07 +0000"  >&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;I&apos;ve uploaded the files from the servers.&lt;/p&gt;

&lt;p&gt;Whilst we kick off the backups at 8am each day, I think the issue starts to appear when we start performing backups of the larger databases on the cluster. Thus the issue appears an hour later then when the backups start. Stopping the backups did resolve the performance issue, which is why the issue appears(may not be) to be when mongodump is running.&lt;/p&gt;

&lt;p&gt;On another cluster we see the issue appear exactly on 8am/8pm.&lt;/p&gt;

&lt;p&gt;Thanks&lt;br/&gt;
Luke&lt;/p&gt;</comment>
                            <comment id="1441048" author="ramon.fernandez" created="Wed, 23 Nov 2016 13:35:52 +0000"  >&lt;p&gt;Thanks for uploading the &lt;tt&gt;diagnostic.data&lt;/tt&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lukemorfitt&quot; class=&quot;user-hover&quot; rel=&quot;lukemorfitt&quot;&gt;lukemorfitt&lt;/a&gt;, we&apos;re looking at it.&lt;/p&gt;

&lt;p&gt;Can you please also upload the compressed &lt;tt&gt;mongod&lt;/tt&gt; logs for all three nodes? The commands graph shows pattern changes at 9pm and 9am, not 8am/8pm; not sure if this is an issue with cacti, a typo in the description, or just a coincidence &amp;#8211; but the logs should be able to tell us more about what&apos;s going on.&lt;/p&gt;</comment>
                            <comment id="1440247" author="lukemorfitt" created="Tue, 22 Nov 2016 17:00:42 +0000"  >&lt;p&gt;Hello Thomas,&lt;/p&gt;

&lt;p&gt;Thank you for the quick response.&lt;/p&gt;

&lt;p&gt;I&apos;ve uploaded the files via Curl.&lt;/p&gt;

&lt;p&gt;To confirm mongodump runs on both secondaries, and never on the master.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Luke&lt;/p&gt;</comment>
                            <comment id="1440188" author="thomas.schubert" created="Tue, 22 Nov 2016 16:17:52 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=lukemorfitt&quot; class=&quot;user-hover&quot; rel=&quot;lukemorfitt&quot;&gt;lukemorfitt&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for reporting this issue. I&apos;ve created a &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/c12466a9-0e58-4ff8-862b-4735666a7707.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;secure portal&lt;/a&gt; for you to use. Files uploaded to this portal are only visible to MongoDB employees investigating the issue and are routinely deleted after some time. Please upload the &lt;tt&gt;diagnostic.data&lt;/tt&gt; for the primary and secondary running mongodump.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="346948">SERVER-27753</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="144612" name="Cluster cpu.png" size="89500" author="lukemorfitt" created="Tue, 22 Nov 2016 13:23:23 +0000"/>
                            <attachment id="144613" name="Master 1 day commands graph .png" size="146207" author="lukemorfitt" created="Tue, 22 Nov 2016 13:20:32 +0000"/>
                            <attachment id="145322" name="Secondary running mongodump.png" size="112874" author="lukemorfitt" created="Mon, 5 Dec 2016 14:02:18 +0000"/>
                            <attachment id="145323" name="master commands.png" size="44449" author="lukemorfitt" created="Mon, 5 Dec 2016 14:02:16 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 22 Nov 2016 16:17:52 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 51 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 51 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>lukemorfitt</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjq5j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hss4yv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hritdz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>