<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:02:35 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3266] mongos consistently locks up distributing parallel updates to multiple shards -- cluster unusable</title>
                <link>https://jira.mongodb.org/browse/SERVER-3266</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;a data set of 120m docs in one collection is being updated to add data to each document.&lt;/p&gt;

&lt;p&gt;every few million records processed (mixed reads and writes, many fewer writes), mongos becomes unresponsive. this number has some down from 20m gradually until it now locks after only a few million.&lt;/p&gt;

&lt;p&gt;queries against each of the shard and config mongod instance show them to be responsive to requests&lt;/p&gt;

&lt;p&gt;queries against mongos hang indefinitely as does db.stats().&lt;/p&gt;

&lt;p&gt;gdb stack trace (attached) shows many threads in mongo::ChunkManager::getShardsForQuery waiting  to obtain mongo::rwlock::rwlock&lt;/p&gt;</description>
                <environment>centos, two machines, four shards each, 30 simultaneous updateres</environment>
        <key id="18317">SERVER-3266</key>
            <summary>mongos consistently locks up distributing parallel updates to multiple shards -- cluster unusable</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.mongodb.org/images/icons/priorities/blocker.svg">Blocker - P1</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="greg_10gen">Greg Studer</assignee>
                                    <reporter username="rgiar">Rob Giardina</reporter>
                        <labels>
                    </labels>
                <created>Wed, 15 Jun 2011 17:28:56 +0000</created>
                <updated>Tue, 12 Jul 2016 00:17:15 +0000</updated>
                            <resolved>Wed, 22 Jun 2011 19:14:33 +0000</resolved>
                                    <version>1.8.1</version>
                    <version>1.8.2</version>
                    <version>1.8.3</version>
                                    <fixVersion>1.8.3</fixVersion>
                                    <component>Concurrency</component>
                    <component>Sharding</component>
                    <component>Stability</component>
                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="38806" author="greg_10gen" created="Wed, 22 Jun 2011 19:14:32 +0000"  >&lt;p&gt;No worries, reopen if you see the problem again.&lt;/p&gt;</comment>
                            <comment id="38763" author="rgiar" created="Wed, 22 Jun 2011 16:22:13 +0000"  >&lt;p&gt;Hi Greg,&lt;/p&gt;

&lt;p&gt;Unfortunately, I had to tear down the sharded config to make progress and I&lt;br/&gt;
am now depending on the system in question. For a real live test, I need to&lt;br/&gt;
simulate it on other systems which will take a while to build.&lt;/p&gt;

&lt;p&gt;Thanks for the speedy fix, you&apos;ve restored my faith in sharding, I&apos;ll come&lt;br/&gt;
back to it soon.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Rob&lt;/p&gt;


</comment>
                            <comment id="38564" author="greg_10gen" created="Tue, 21 Jun 2011 18:46:23 +0000"  >&lt;p&gt;just pinging for an update on your status...&lt;/p&gt;</comment>
                            <comment id="38127" author="greg_10gen" created="Fri, 17 Jun 2011 23:05:27 +0000"  >&lt;p&gt;pretty much - the codepath usually works, but pretty sure interleaved writes can cause issues.  Fix didn&apos;t make it into 1.8.2 unfortunately, but if you grab that checkout version it should be exactly 1.8.2+the patch.  &lt;br/&gt;
I&apos;ll close if/when we confirm things on your end.&lt;/p&gt;</comment>
                            <comment id="38118" author="rgiar" created="Fri, 17 Jun 2011 21:53:36 +0000"  >&lt;p&gt;thx for the workaround, i&apos;ve had to decommission the cluster and move to a single instance for the moment so i can&apos;t test this now. your fix seems pretty definitive &amp;#8211; I didn&apos;t read the surrounding code but the diff looks like you&apos;re not trying to get a (non-reentrant?) lock. I&apos;m very optimistic.&lt;/p&gt;</comment>
                            <comment id="38115" author="auto" created="Fri, 17 Jun 2011 21:47:10 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{u&apos;login&apos;: u&apos;gregstuder&apos;, u&apos;name&apos;: u&apos;gregs&apos;, u&apos;email&apos;: u&apos;greg@10gen.com&apos;}
&lt;p&gt;Message: don&apos;t reacquire read lock when getting all shards &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-3266&quot; title=&quot;mongos consistently locks up distributing parallel updates to multiple shards -- cluster unusable&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-3266&quot;&gt;&lt;del&gt;SERVER-3266&lt;/del&gt;&lt;/a&gt;&lt;br/&gt;
Branch: v1.8&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1f9df58e76cd47a19475c7532b114e4ec55af9b5&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1f9df58e76cd47a19475c7532b114e4ec55af9b5&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="38063" author="greg_10gen" created="Fri, 17 Jun 2011 16:09:51 +0000"  >&lt;p&gt;Is your shard key included in each of the queries?  One potential workaround may be to always ensure that the (first part of) your shard key is explicitly bounded in your queries by a min or max value ( an actual value, not $Min/$MaxKey ).  key : &lt;/p&gt;
{ $gt : -1000000 }
&lt;p&gt; for example.&lt;/p&gt;</comment>
                            <comment id="37902" author="greg_10gen" created="Thu, 16 Jun 2011 18:17:54 +0000"  >&lt;p&gt;Ah yes, missed it, thanks.&lt;/p&gt;</comment>
                            <comment id="37900" author="rgiar" created="Thu, 16 Jun 2011 17:56:03 +0000"  >&lt;p&gt;I sent a tar file via private email to you and Eliot. Did you get them?&lt;/p&gt;</comment>
                            <comment id="37899" author="greg_10gen" created="Thu, 16 Jun 2011 17:52:52 +0000"  >&lt;p&gt;Very understandable &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;  Trying a 1.9 mongos with v1.8 mongod may be less risky since it&apos;s only routing your requests, not storing data - though again, this is not a configuration we test.&lt;/p&gt;

&lt;p&gt;Do you have the logs for the mongos process from before and after the slowdown and eventual hang?&lt;/p&gt;</comment>
                            <comment id="37877" author="rgiar" created="Thu, 16 Jun 2011 15:50:35 +0000"  >&lt;p&gt;this is production data &amp;#8211; only using the 1.8.x versions; i consider the odd numbered versions a subtle invitation to lose data &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="37868" author="greg_10gen" created="Thu, 16 Jun 2011 15:05:06 +0000"  >&lt;p&gt;are you seeing this in a dev environment or production system?  The need for locking in some of these places has been removed in v1.9.  &lt;/p&gt;</comment>
                            <comment id="37693" author="eliot" created="Wed, 15 Jun 2011 17:59:12 +0000"  >&lt;p&gt;Looks like chunk updates are slow.&lt;/p&gt;

&lt;p&gt;Can you attach mongos log?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="12082" name="mongos-thread-dump.txt" size="99888" author="rgiar" created="Wed, 15 Jun 2011 17:28:56 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 15 Jun 2011 17:59:12 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 years, 35 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 years, 35 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>auto</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>greg_10gen</customfieldvalue>
            <customfieldvalue>rgiar</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hroxrj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrirjz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>23117</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs39tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>