<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:15:57 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-7922] All operations blocked on one sharded collection</title>
                <link>https://jira.mongodb.org/browse/SERVER-7922</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Every morning since last week, all operations to a sharded collection are failing.&lt;/p&gt;

&lt;p&gt;Here are application side errors :&lt;br/&gt;
setShardVersion failed host: mdbcis4-01-sv.criteo.prod:27021 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), ns: &quot;counters.statistics&quot;, version: Timestamp 4000|3, versionEpoch: ObjectId(&apos;000000000000000000000000&apos;), globalVersion: Timestamp 6000|0, globalVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), reloadConfig: true, errmsg: &quot;shard global version for collection is higher than trying to set to &apos;counters.statistics&apos;&quot;, ok: 0.0 }
&lt;p&gt;&apos;. (Response was { &quot;err&quot; : &quot;setShardVersion failed host: mdbcis4-01-sv.criteo.prod:27021 &lt;/p&gt;
{ oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), ns: \&quot;counters.statistics\&quot;, version: Timestamp 4000|3, versionEpoch: ObjectId(&apos;000000000000000000000000&apos;), globalVersion: Timestamp 6000|0, globalVersionEpoch: ObjectId(&apos;000000000000000000000000&apos;), reloadConfig: true, errmsg: \&quot;shard global version for collection is higher than trying to set to &apos;counters.statistics&apos;\&quot;, ok: 0.0 }
&lt;p&gt;&quot;, &quot;code&quot; : 10429, &quot;n&quot; : 0, &quot;ok&quot; : 1.0 }). : MongoDB.Driver.SafeModeResult SendMessage(MongoDB.Driver.Internal.MongoRequestMessage, MongoDB.Driver.SafeMode)&lt;/p&gt;

&lt;p&gt;Server-side ones :&lt;br/&gt;
warning: aborted moveChunk because official version less than mine?: official 5|1||000000000000000000000000 mine: 6|0||000000000000000000000000&lt;/p&gt;

&lt;p&gt;Restarting mongod unlocks operations until next morning.&lt;/p&gt;

&lt;p&gt;In attachment, logs of servers involved in the moveChunk process (shard4 to shard6), sh_status output, changelog collection output&lt;br/&gt;
In logs issue starts at Wed Dec 12 06:47:09, ends at Wed Dec 12 09:30:00 after restart.&lt;/p&gt;

&lt;p&gt;Link to our MMS dashboard : &lt;a href=&quot;https://mms.10gen.com/host/list/4f8d732587d1d86fa8b99c12&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://mms.10gen.com/host/list/4f8d732587d1d86fa8b99c12&lt;/a&gt;&lt;br/&gt;
Problem occured before we added 8th shard and seems to be linked to previous bugs : &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7034&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-7034&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-7821&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/SERVER-7821&lt;/a&gt;&lt;/p&gt;</description>
                <environment>Linux Centos5/6</environment>
        <key id="59270">SERVER-7922</key>
            <summary>All operations blocked on one sharded collection</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="david.hows">David Hows</assignee>
                                    <reporter username="k.hodin@criteo.com">Kl&#233;bert Hodin</reporter>
                        <labels>
                    </labels>
                <created>Thu, 13 Dec 2012 13:05:47 +0000</created>
                <updated>Mon, 11 Jul 2016 17:56:29 +0000</updated>
                            <resolved>Wed, 23 Jan 2013 23:57:10 +0000</resolved>
                                    <version>2.2.2</version>
                                                    <component>Performance</component>
                    <component>Sharding</component>
                    <component>Stability</component>
                                        <votes>1</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="324035" author="ian@10gen.com" created="Mon, 29 Apr 2013 14:56:46 +0000"  >&lt;p&gt;Hi Javi, please open a new ticket in the SERVER project with any further details you can include.&lt;/p&gt;</comment>
                            <comment id="323864" author="javi@nuroa.com" created="Mon, 29 Apr 2013 08:21:36 +0000"  >&lt;p&gt;Hi!&lt;/p&gt;

&lt;p&gt;during the last 2 weeks I have had the same problem 4 times with the same mongoDB version... Removing data and re-syncing is the only way to solve this problem? Is there any other thing that I can do to solve it without re-syncing?&lt;/p&gt;

&lt;p&gt;Regards,&lt;/p&gt;</comment>
                            <comment id="247326" author="k.hodin@criteo.com" created="Wed, 23 Jan 2013 07:25:08 +0000"  >&lt;p&gt;I don&apos;t think so.&lt;br/&gt;
I&apos;ll open a new bug if necessary.&lt;/p&gt;

&lt;p&gt;Kl&#233;bert&lt;/p&gt;</comment>
                            <comment id="247160" author="david.hows" created="Wed, 23 Jan 2013 02:18:18 +0000"  >&lt;p&gt;Hi Klebert,&lt;/p&gt;

&lt;p&gt;Is there anything more we can do for you on this issue?&lt;/p&gt;

&lt;p&gt;It seems your re-sync has solved the problem and the older logs were unavailable.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="236996" author="k.hodin@criteo.com" created="Fri, 11 Jan 2013 08:39:44 +0000"  >&lt;p&gt;I mean no issue since we resync&apos;d primaries mongod of shards 4 and 6.&lt;/p&gt;</comment>
                            <comment id="236810" author="david.hows" created="Fri, 11 Jan 2013 03:16:08 +0000"  >&lt;p&gt;Hi Klebert,&lt;/p&gt;

&lt;p&gt;What data did you remove? Or do you just mean the resync?&lt;/p&gt;</comment>
                            <comment id="236064" author="k.hodin@criteo.com" created="Thu, 10 Jan 2013 13:17:13 +0000"  >&lt;p&gt;No it didn&apos;t.&lt;br/&gt;
It seems removing data and restarting mongod fixed this issue.&lt;/p&gt;</comment>
                            <comment id="235793" author="david.hows" created="Thu, 10 Jan 2013 05:11:15 +0000"  >&lt;p&gt;Hi Klebert,&lt;/p&gt;

&lt;p&gt;Has this issue continued occurring since your restart and has it continued subsequently?&lt;/p&gt;

&lt;p&gt;If so, can you attach a log for one of the instances that this issue occurs on from the time of the restart till after you have seen the issue occur?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="226031" author="k.hodin@criteo.com" created="Fri, 28 Dec 2012 11:02:56 +0000"  >&lt;p&gt;Hi David,&lt;/p&gt;

&lt;p&gt;It happened 15 days ago, we no longer have these log files.&lt;/p&gt;

&lt;p&gt;Kl&#233;bert&lt;/p&gt;</comment>
                            <comment id="225852" author="david.hows" created="Fri, 28 Dec 2012 04:24:02 +0000"  >&lt;p&gt;Hi Klebert,&lt;/p&gt;

&lt;p&gt;Can you attach the logs for the 24hrs following the restart? &lt;/p&gt;

&lt;p&gt;It should show the initial kickoff of the migration that failed that we were unable to see in the last set of logs.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="224417" author="k.hodin@criteo.com" created="Wed, 26 Dec 2012 09:07:38 +0000"  >&lt;p&gt;Yes we tried. It only fixed the issue for 24h (until it happened again the morning after).&lt;/p&gt;</comment>
                            <comment id="223802" author="eliot" created="Tue, 25 Dec 2012 07:59:10 +0000"  >&lt;p&gt;Have you tried restart mongod on mdbcis4-01-sv.criteo.prod:27021?&lt;/p&gt;</comment>
                            <comment id="223266" author="k.hodin@criteo.com" created="Mon, 24 Dec 2012 10:23:12 +0000"  >&lt;p&gt;Hi David,&lt;/p&gt;

&lt;p&gt;Any updates on this issue ?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;Kl&#233;bert&lt;/p&gt;</comment>
                            <comment id="219106" author="kamaradclimber" created="Wed, 19 Dec 2012 09:58:30 +0000"  >&lt;p&gt;Hi David, &lt;/p&gt;

&lt;p&gt;resyncing servers (&lt;a href=&quot;http://docs.mongodb.org/manual/administration/replica-sets/#resyncing-a-member-of-a-replica-set&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/administration/replica-sets/#resyncing-a-member-of-a-replica-set&lt;/a&gt;) is a hammer to catch a fly but solves most weird in mongodb so we use it quite often.&lt;/p&gt;</comment>
                            <comment id="219105" author="k.hodin@criteo.com" created="Wed, 19 Dec 2012 09:58:12 +0000"  >&lt;p&gt;&lt;a href=&quot;http://docs.mongodb.org/manual/administration/replica-sets/#replica-set-auto-resync-stale-member&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://docs.mongodb.org/manual/administration/replica-sets/#replica-set-auto-resync-stale-member&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="218959" author="david.hows" created="Wed, 19 Dec 2012 05:23:04 +0000"  >&lt;p&gt;Thanks Kl&#233;bert,&lt;/p&gt;

&lt;p&gt;I&apos;m just trying to follow what occurred to cause those versioning errors. &lt;/p&gt;

&lt;p&gt;What do you mean by full-resync&apos;d?&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="218107" author="k.hodin@criteo.com" created="Tue, 18 Dec 2012 09:30:35 +0000"  >&lt;p&gt;Hi David, &lt;/p&gt;

&lt;p&gt;Here&apos;s a dump config dbs.&lt;br/&gt;
We don&apos;t you have any logs prior to Fri Dec 7 04:02:19.&lt;/p&gt;

&lt;p&gt;This bug didn&apos;t show up again since we full resynced shards 4 and 6.&lt;/p&gt;</comment>
                            <comment id="217983" author="david.hows" created="Tue, 18 Dec 2012 05:33:42 +0000"  >&lt;p&gt;Hi Kl&#233;bert, Gr&#233;goire,&lt;/p&gt;

&lt;p&gt;Thanks for the logs. I&apos;ve been looking through these extra logs and can see a few interesting things in the logs relating to how shard versions are calculated.&lt;/p&gt;

&lt;p&gt;I&apos;d like to confirm what is going on within your config servers, can you please dump the config database from all three of your config servers and attach it to the ticket?&lt;/p&gt;

&lt;p&gt;Additionally, do you have any logs prior to &lt;tt&gt;Fri Dec 7 04:02:19&lt;/tt&gt;? &lt;/p&gt;

&lt;p&gt;Thanks,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="217039" author="k.hodin@criteo.com" created="Mon, 17 Dec 2012 09:47:51 +0000"  >&lt;p&gt;Older logs from shard4.&lt;/p&gt;</comment>
                            <comment id="214766" author="kamaradclimber" created="Fri, 14 Dec 2012 07:44:57 +0000"  >&lt;p&gt;I&apos;ll give you all logs and things on Monday&lt;/p&gt;</comment>
                            <comment id="214765" author="kamaradclimber" created="Fri, 14 Dec 2012 07:44:30 +0000"  >&lt;p&gt;Hello David,  &lt;br/&gt;
could you request everything you need at once &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/wink.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="214689" author="david.hows" created="Fri, 14 Dec 2012 05:38:04 +0000"  >&lt;p&gt;Hi Gr&#233;goire,&lt;/p&gt;

&lt;p&gt;I&apos;ve been following up on this with your logs from the other ticket.&lt;/p&gt;

&lt;p&gt;Would it be possible to get some of earlier logs from shard4? Currently the log starts at &lt;tt&gt;Wed Dec 12 04:02:17&lt;/tt&gt; and is in the middle of a chunk migration. I would like to see what happens at the start of this migration.&lt;/p&gt;

&lt;p&gt;Cheers,&lt;/p&gt;

&lt;p&gt;David&lt;/p&gt;</comment>
                            <comment id="214058" author="kamaradclimber" created="Thu, 13 Dec 2012 15:41:17 +0000"  >&lt;p&gt;No, this collection is upsert only (and some reads of course).&lt;br/&gt;
No backup restoring&lt;/p&gt;</comment>
                            <comment id="214046" author="scotthernandez" created="Thu, 13 Dec 2012 15:21:29 +0000"  >&lt;p&gt;Are you dropping the collection and recreating it at any point? Or restoring from a backup of the whole cluster?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="58172">SERVER-7821</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="50453">SERVER-7034</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="22465" name="changelog" size="11339074" author="k.hodin@criteo.com" created="Thu, 13 Dec 2012 13:05:47 +0000"/>
                            <attachment id="22631" name="config_dump.tar" size="37990400" author="k.hodin@criteo.com" created="Tue, 18 Dec 2012 09:30:35 +0000"/>
                            <attachment id="22466" name="mongod-shard4-shard6-logs.zip" size="1447231" author="k.hodin@criteo.com" created="Thu, 13 Dec 2012 13:05:47 +0000"/>
                            <attachment id="22568" name="mongod-shard4_20121211.zip" size="1554628" author="k.hodin@criteo.com" created="Mon, 17 Dec 2012 09:47:51 +0000"/>
                            <attachment id="22467" name="sh_status" size="6125" author="k.hodin@criteo.com" created="Thu, 13 Dec 2012 13:05:47 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>24.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 13 Dec 2012 15:21:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        10 years, 42 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            10 years, 42 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10020"><![CDATA[Linux]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>kamaradclimber</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>javi@nuroa.com</customfieldvalue>
            <customfieldvalue>k.hodin@criteo.com</customfieldvalue>
            <customfieldvalue>scotthernandez</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrnepz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrl2iv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>36729</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsw13b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>