<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:28:29 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-12415] spikes in write lock % only on one out of 5 shards</title>
                <link>https://jira.mongodb.org/browse/SERVER-12415</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Since a few days ago (January 14th) we&apos;re seeing spikes in write lock % tracked in MMS, but only on one shard out of 5 (our _c shard). We&apos;re also seeing spikes in read queue at the same time write lock % spikes.&lt;br/&gt;
There are a few non-sharded collections having this shard as their primary, but they don&apos;t have that many writes.&lt;/p&gt;

&lt;p&gt;However, we don&apos;t see spikes in IO / CPU load or network traffic on those servers, so we don&apos;t understand the exact cause of those spikes.&lt;br/&gt;
Can anybody explain what could be the bottleneck other than disk IO, CPU, or network bandwidth?&lt;/p&gt;</description>
                <environment>CentOS 6.4, sharded cluster with 5 shards</environment>
        <key id="106899">SERVER-12415</key>
            <summary>spikes in write lock % only on one out of 5 shards</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="stephen.steneker@mongodb.com">Stennie Steneker</assignee>
                                    <reporter username="pcmaniac">Nemanja Dubravac</reporter>
                        <labels>
                    </labels>
                <created>Tue, 21 Jan 2014 11:39:08 +0000</created>
                <updated>Mon, 16 Feb 2015 15:21:38 +0000</updated>
                            <resolved>Mon, 16 Feb 2015 15:21:38 +0000</resolved>
                                    <version>2.4.8</version>
                                                    <component>Concurrency</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="518951" author="stennie" created="Wed, 19 Mar 2014 02:04:23 +0000"  >&lt;p&gt;Hi Nemanja,&lt;/p&gt;

&lt;p&gt;Thank you for following up with a summary of the root cause of your lock percentage spikes .. I will close out this issue.&lt;/p&gt;

&lt;p&gt;Large embedded arrays with frequent updates can definitely lead to performance problems. A related blog write up from one of our solution architects includes some considerations: &quot;&lt;a href=&quot;http://askasya.com/post/largeembeddedarrays&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Why shouldn&apos;t I embed large arrays in my documents?&lt;/a&gt;&quot;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Stephen&lt;/p&gt;</comment>
                            <comment id="518936" author="pcmaniac" created="Wed, 19 Mar 2014 00:50:24 +0000"  >&lt;p&gt;Hi Stephen,&lt;/p&gt;

&lt;p&gt;I&apos;m sorry for not following up on this issue right after we figured what was causing it. Thanks for reminding me of it and hopefully if someone else comes up with the same problem, they will be able to find the following answer:&lt;/p&gt;

&lt;p&gt;&lt;b&gt;One of our user objects had a key with a huge array value (&amp;gt; 1K elements), and the key was indexed. It&apos;s an amateur mistake to have arrays indexed if their size is not limited, but there you have it. Whenever this user was online and active, or some other operation triggered an insert into this growing array, it caused a spike in lock % on the shard where this user object was located.&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;I really hope this will help someone else, because we wasted a lot of hours checking whether sharding was set up correctly, whether balancing was working, whether there were hardware issues... and so on.&lt;/p&gt;

&lt;p&gt;Anyway, thank you Stephen for checking this issue and trying to help.&lt;/p&gt;</comment>
                            <comment id="518145" author="stennie" created="Tue, 18 Mar 2014 05:28:47 +0000"  >&lt;p&gt;Hi Nemanja,&lt;/p&gt;

&lt;p&gt;Aside from chunk migrations, some other possibilities for increased write lock % on specific shards can include:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;write activity for &lt;a href=&quot;http://docs.mongodb.org/manual/faq/sharding/#what-happens-to-unsharded-collections-in-sharded-databases&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;unsharded collections&lt;/a&gt; (which will live on a primary shard for each database)&lt;/li&gt;
	&lt;li&gt;a poor &lt;a href=&quot;http://docs.mongodb.org/manual/core/sharding-shard-key/#write-scaling&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;shard key&lt;/a&gt; - for example, a monotonically increasing value (such as ObjectID) can create a &quot;hot shard&quot;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If you would be willing to upload a dump of your &lt;tt&gt;config&lt;/tt&gt; database for review, we could have a look at the migration history and primary shards for unsharded collections. Please confirm if you&apos;d like to do so &amp;#8211; we can provide a private SCP dropbox so the information is not public on this issue.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Stephen&lt;/p&gt;</comment>
                            <comment id="485474" author="pcmaniac" created="Wed, 22 Jan 2014 09:31:01 +0000"  >&lt;p&gt;There are no chunk migrations at the time of those spikes as balancing is scheduled for night time.&lt;br/&gt;
I can see which collections are active, but I don&apos;t understand what is causing the write lock to jump to 60% and several hundred reads are queued if there are no hardware bottlenecks visible.&lt;br/&gt;
Can you tell me what else can affect the write lock % other than the mentioned hardware resources?&lt;/p&gt;

&lt;p&gt;Here&apos;s a concrete example of such spikes. Under normal behavior (even while having a lot of users) our total backend response time is below 250ms and write lock % is below 10%. However, at random times throughout the whole day we see spikes in write lock % where they reach 60% or more for a short period and then our total response time jumps to several seconds, as you can see in the attached screenshots.&lt;/p&gt;</comment>
                            <comment id="484805" author="dan@10gen.com" created="Tue, 21 Jan 2014 13:20:52 +0000"  >&lt;p&gt;Are there chunk migrations to or from that shard?  You should be able to get an idea of activity by looking at the logs for that shard or running &lt;tt&gt;mongotop&lt;/tt&gt; to get an idea of which collections are active.&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="36514" name="backend_response_time.png" size="51516" author="pcmaniac" created="Wed, 22 Jan 2014 09:31:24 +0000"/>
                            <attachment id="36515" name="shard_c_write_lock_percentage.png" size="19452" author="pcmaniac" created="Wed, 22 Jan 2014 09:31:24 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 21 Jan 2014 13:20:52 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 48 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 48 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>pcmaniac</customfieldvalue>
            <customfieldvalue>stephen.steneker@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrm3jj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrwlif:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>104219</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsozqf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>