<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:16:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-70568] Latency spikes without any additional load</title>
                <link>https://jira.mongodb.org/browse/SERVER-70568</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We have a MongoDB 4.4.10 sharded cluster with 10 shards with PSS topology. From time to time (2 or 3 times a day) we encounter huge latency spike that can&apos;t be explained by queries our cluster serving currently. At that moment CPU load on problematic shard primary plummets to near zero. We observe no load increase on IO either. Everything that is currently working on this MongoDB instance starts to perform slowly. Such diagnostic tools as `atop` and `telegraf` also suffer from latency increase. `atop` loses a segment of time at the moment.&lt;/p&gt;

&lt;p&gt;We run our cluster on AWS EC2 instances but CloudWatch metrics regarding instance health are quiet. We recently upscaled instances in our cluster x2 from r5d.4xlarge to r5d.8xlarge but the problem still persists and its frequency hasn&apos;t lowered. We have another MongoDB clusters and other clusters are performing well.&lt;/p&gt;

&lt;p&gt;I attached an example FTDC log at the moment of latency spike. The exact time is 09:03-09:12 UTC, you can find it by connection count spike.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2159149">SERVER-70568</key>
            <summary>Latency spikes without any additional load</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="yuan.fang@mongodb.com">Yuan Fang</assignee>
                                    <reporter username="sz">Sergey Zagursky</reporter>
                        <labels>
                    </labels>
                <created>Fri, 14 Oct 2022 11:22:45 +0000</created>
                <updated>Mon, 31 Oct 2022 19:31:47 +0000</updated>
                            <resolved>Mon, 31 Oct 2022 19:31:47 +0000</resolved>
                                    <version>4.4.10</version>
                                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="4941886" author="JIRAUSER1270794" created="Mon, 31 Oct 2022 19:31:07 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Yuan&lt;/p&gt;</comment>
                            <comment id="4908105" author="JIRAUSER1270794" created="Mon, 17 Oct 2022 19:31:50 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sz&quot; class=&quot;user-hover&quot; rel=&quot;sz&quot;&gt;sz&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for reporting this issue. After discussing with the Server Triage team, I don&apos;t think this issue is &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31417&quot; title=&quot;Improve tcmalloc when decommitting large amounts of memory&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-31417&quot;&gt;SERVER-31417&lt;/a&gt; because no significant amounts of memory &lt;tt&gt;tcmalloc&lt;/tt&gt; de-committing were observed during the latency spikes. Unfortunately, I haven&apos;t been able to spot a clear cause and didn&apos;t find any performance bottlenecks, but it would be worth verifying the secondary node, and the mongodb.log. Additionally, it&apos;s not clear from the data so far that &lt;tt&gt;tcmallocReleaseRate=0&lt;/tt&gt; would help, and we do not typically recommend config &lt;tt&gt;tcmallocReleaseRate=0&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;With all that being said, we still need additional information to diagnose the problem.&lt;/p&gt;

&lt;p&gt;I&apos;ve created a secure &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNzgyMzE3NzYyMDUiLCJleHAiOjE2Njg4NzM3ODF9.Xt6HbZFx_cTdbAYxdI8YwhCrHFP-g7MDBvh84gOZEu8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;upload portal&lt;/a&gt; for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;For each node (including both the primary and secondary nodes) in the replica set, would you please archive (tar or zip) and upload to that link:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Full mongod.log and diagnostic.data include the spikes of latencies, with &lt;tt&gt;tcmallocReleaseRate=1&lt;/tt&gt;(default).&lt;/li&gt;
	&lt;li&gt;Full mongod.log and diagnostic.data for a time when you believe the latencies should have occurred but did not,&#160;with default &lt;tt&gt;tcmallocReleaseRate=0.&lt;/tt&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Regards,&lt;br/&gt;
Yuan&lt;/p&gt;</comment>
                            <comment id="4906359" author="sz" created="Mon, 17 Oct 2022 09:36:29 +0000"  >&lt;p&gt;As for now, after 48 hours, I don&apos;t see any problems with `tcmallocReleaseRate` set to 0. What long term problems could this setting inflict?&lt;/p&gt;</comment>
                            <comment id="4906356" author="sz" created="Mon, 17 Oct 2022 09:34:56 +0000"  >&lt;p&gt;There is a relevant issue: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-31417&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-31417&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="4906311" author="sz" created="Mon, 17 Oct 2022 09:00:31 +0000"  >&lt;p&gt;I have additional input on our problem. It seems that setting `tcmallocReleaseRate` to 0 effectively stops our latency spikes from happening again. Therefore I suspect we have a pathological workload that makes tcmalloc to behave this way. What could it be?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="408315" name="metrics.2022-10-13T08-12-17Z-00000" size="5648479" author="sz" created="Fri, 14 Oct 2022 11:32:47 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 14 Oct 2022 21:48:42 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 14 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>yuan.fang@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 14 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>sz</customfieldvalue>
            <customfieldvalue>yuan.fang@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1e2v3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0wqts:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[yuan.fang@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i1dp0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>