<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:11:20 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26171] Gradual Degradation of Performance over days</title>
                <link>https://jira.mongodb.org/browse/SERVER-26171</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Wired Tiger performance degrades over time:&lt;/p&gt;

&lt;p&gt;I&apos;ve attached a graph of CPU &quot;idle&quot; time.&lt;/p&gt;

&lt;p&gt;The first 2/3 of the attached graph shows our database primaries working harder each day. The actual number of DB Ops is generally stable over this time period.&lt;/p&gt;

&lt;p&gt;On September 10th, I bounce the primaries to a separate set of servers. These new servers &lt;em&gt;also&lt;/em&gt; experience gradual degradation, and the previous primaries (now secondaries) continue to have less idle time compared to the new primaries, and the other set of secondaries.&lt;/p&gt;

&lt;p&gt;On September 13th, I upgraded the servers to larger instances, and reboot a bunch of servers. After a reboot, all servers start behaving normally, but the new masters are now starting to slow down.&lt;/p&gt;

&lt;p&gt;(1) Has anyone seen something like this before?&lt;br/&gt;
(2) How can I help track this down?&lt;br/&gt;
-Mike&lt;/p&gt;</description>
                <environment>Linux on AWS - Sharded Replica Set - 2 Shards, 3-node Replica Sets</environment>
        <key id="317060">SERVER-26171</key>
            <summary>Gradual Degradation of Performance over days</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="tewner">Michael Tewner</reporter>
                        <labels>
                    </labels>
                <created>Mon, 19 Sep 2016 08:12:29 +0000</created>
                <updated>Fri, 6 Jan 2017 19:12:01 +0000</updated>
                            <resolved>Fri, 6 Jan 2017 19:12:01 +0000</resolved>
                                    <version>3.0.12</version>
                                                                        <votes>1</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="1471434" author="thomas.schubert" created="Fri, 6 Jan 2017 19:12:01 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tewner&quot; class=&quot;user-hover&quot; rel=&quot;tewner&quot;&gt;tewner&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. If this is still an issue for you after upgrading, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1437873" author="thomas.schubert" created="Fri, 18 Nov 2016 18:59:06 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tewner&quot; class=&quot;user-hover&quot; rel=&quot;tewner&quot;&gt;tewner&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=may&quot; class=&quot;user-hover&quot; rel=&quot;may&quot;&gt;may&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We have not been able to identify the root cause of the gradual change in cpu load. My understanding is that this does not impact the performance of the cluster, as the number database operations remains consistent.  If this is still an issue for you, would you please consider upgrading to MongoDB 3.2 or test the latest MongoDB 3.4 release candidate? These branches contain many improvements, which may significantly impact the behavior you are observing&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1402112" author="may" created="Thu, 6 Oct 2016 16:09:01 +0000"  >&lt;p&gt;Hi Thomas Schubert,&lt;/p&gt;

&lt;p&gt;we are doing a weekly backup of mongo. we took a look at grafana and it seemed that a day after the backup the servers has less load then usual, and every day until the next backup there is more and more load. i attached a printscreen of the graph of cpu idle in time range of 21 days.&lt;br/&gt;
Furthermore we looked at the graph of the dirty cache in the exact same time range and saw that if relatively stable(also attached).&lt;/p&gt;

&lt;p&gt;Thank you!&lt;/p&gt;</comment>
                            <comment id="1395050" author="thomas.schubert" created="Tue, 27 Sep 2016 20:30:59 +0000"  >&lt;p&gt;Thanks for uploading the logs, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=may&quot; class=&quot;user-hover&quot; rel=&quot;may&quot;&gt;may&lt;/a&gt;. We&apos;re still investigating this issue and will update this ticket when we know more.&lt;/p&gt;</comment>
                            <comment id="1390664" author="may" created="Thu, 22 Sep 2016 11:01:14 +0000"  >&lt;p&gt;Hi, i attached the logs with the relevant output.&lt;/p&gt;

&lt;p&gt;Thank you, May. &lt;/p&gt;</comment>
                            <comment id="1389489" author="thomas.schubert" created="Wed, 21 Sep 2016 03:49:30 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=may&quot; class=&quot;user-hover&quot; rel=&quot;may&quot;&gt;may&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for providing the output of the command. I see that the maximum tree depth of this collection is 3, which indicates that this collection is not affected by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21063&quot; title=&quot;MongoDB with WiredTiger can build very deep trees&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21063&quot;&gt;&lt;del&gt;SERVER-21063&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Unfortunately, to continue to investigate, we will need to collect additional metrics. If you are unable to upgrade to MongoDB 3.2.9 as I suggested, would you please run the following script on your Primary?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:true}))); sleep(1000*${delay:?})}&quot; &amp;gt;ss.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x ${delay:?} &amp;gt;iostat.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will collect much of the same information that is captured by later version of MongoDB, and will allow us to continue to investigate this behavior. Please leave this script running for a day before uploading the files so we can see the performance degradation in the stats.&lt;/p&gt;

&lt;p&gt;Thank you for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1388436" author="may" created="Tue, 20 Sep 2016 10:04:19 +0000"  >&lt;p&gt;Hi, I&apos;ve attached a file of the &quot;db.collection.stats(&lt;/p&gt;
{indexDetails:true}
&lt;p&gt;)&quot; command output. The file contain output for both shards.&lt;br/&gt;
Thank you, May.&lt;/p&gt;</comment>
                            <comment id="1388358" author="thomas.schubert" created="Tue, 20 Sep 2016 05:19:51 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tewner&quot; class=&quot;user-hover&quot; rel=&quot;tewner&quot;&gt;tewner&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It is possible that the behavior you are observing may be explained by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21063&quot; title=&quot;MongoDB with WiredTiger can build very deep trees&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21063&quot;&gt;&lt;del&gt;SERVER-21063&lt;/del&gt;&lt;/a&gt;. If this is the case, I would recommend performing an initial sync to create balanced trees. &lt;/p&gt;

&lt;p&gt;To determine if your data files are affected by this bug, please provide the following information:&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Please confirm which versions of MongoDB have run on top of your data files.&lt;/li&gt;
	&lt;li&gt;Please provide the output of the following command against a collection that is accessed by your workload: &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.collection.stats({indexDetails:true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1387940" author="thomas.schubert" created="Mon, 19 Sep 2016 19:40:15 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tewner&quot; class=&quot;user-hover&quot; rel=&quot;tewner&quot;&gt;tewner&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for reporting this behavior. Please note that there have been significant improvements to the performance of WiredTiger in MongoDB 3.2. In addition, MongoDB 3.2 includes &lt;a href=&quot;https://docs.mongodb.com/manual/release-notes/3.2/#diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;full time diagnostic capture&lt;/a&gt;, which would help us to identify the cause of this behavior.&lt;/p&gt;

&lt;p&gt;Are you able to upgrade to the latest version of MongoDB to see if it resolves the issue?&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1387271" author="tewner" created="Mon, 19 Sep 2016 08:16:05 +0000"  >&lt;p&gt;Note that the &quot;arb01&quot; servers on the graph are full data nodes.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="138954" name="Screen Shot 2016-09-19 at 11.05.49 AM.png" size="89820" author="tewner" created="Mon, 19 Sep 2016 08:12:29 +0000"/>
                            <attachment id="140786" name="Screen Shot 2016-10-06 at 7.10.12 PM.png" size="214329" author="may" created="Thu, 6 Oct 2016 16:11:20 +0000"/>
                            <attachment id="140787" name="Screen Shot 2016-10-06 at 7.10.36 PM.png" size="79750" author="may" created="Thu, 6 Oct 2016 16:11:20 +0000"/>
                            <attachment id="139119" name="mongo.txt" size="1433" author="may" created="Tue, 20 Sep 2016 10:02:36 +0000"/>
                            <attachment id="139348" name="shard1iostat.log" size="84088734" author="may" created="Thu, 22 Sep 2016 11:01:10 +0000"/>
                            <attachment id="139349" name="shard1ss.tar.gz" size="54471345" author="may" created="Thu, 22 Sep 2016 11:01:10 +0000"/>
                            <attachment id="139350" name="shard2iostat.log" size="83934626" author="may" created="Thu, 22 Sep 2016 11:01:10 +0000"/>
                            <attachment id="139351" name="shard2ss.tar.gz" size="51579513" author="may" created="Thu, 22 Sep 2016 11:01:10 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 19 Sep 2016 19:40:15 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 5 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 5 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>may</customfieldvalue>
            <customfieldvalue>tewner</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjvrr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hspvpz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsejaf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>