<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:24:56 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-30768] Primary queries using maxTimeMS cause temporary shard write unavailability if ExceededTimeLimit</title>
                <link>https://jira.mongodb.org/browse/SERVER-30768</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;b&gt;Setup:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Sharded cluster with replica set shards. MongoDB v3.4.6. WiredTiger with snappy.&lt;br/&gt;
 Collection X exists only on 1 shard (not sharded, probably not relevant).&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Problem:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;When a query fails due to a max time MS timeout (which happens now and again, since we are using a fairly tight limit), MongoS marks the node as failed. (This is incorrect.. the node is NOT failed).&lt;/p&gt;

&lt;p&gt;&lt;b&gt;Result:&lt;/b&gt;&lt;/p&gt;

&lt;p&gt;Since the query was against the primary, and the primary is marked as failed, subsequent write operations fail due to unavailability of the primary. This lasts for a second or a few, presumably until MongoS heartbeat monitor detects the primary as up.&lt;/p&gt;

&lt;p&gt;This renders $maxTimeMS dangerous to use when making primary-side queries--any timed out query will temporarily make the shard write-unavailable.&lt;/p&gt;

&lt;p&gt;Furthermore, it seems that architecturally it&apos;s wrong to have the MongoS mark the host as failed, but then not trigger a failover. This means that the MongoS &quot;failed primary&quot; logic is completely disconnected from the actual primary/replica failover/election logic. This means that when MongoS reports &quot;no primary found&quot; for a shard, it&apos;s not because there&apos;s actually no primary in that replica set! (there is a primary and it&apos;s healthy)&lt;/p&gt;

&lt;p&gt;(I think that this problem applies to queries that hit replicas as well, where the replica is marked as failed, but I haven&apos;t specifically tested that.)&lt;/p&gt;</description>
                <environment></environment>
        <key id="418625">SERVER-30768</key>
            <summary>Primary queries using maxTimeMS cause temporary shard write unavailability if ExceededTimeLimit</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="jack.mulrow@mongodb.com">Jack Mulrow</assignee>
                                    <reporter username="oleg@evergage.com">Oleg Rekutin</reporter>
                        <labels>
                            <label>bkp</label>
                    </labels>
                <created>Mon, 21 Aug 2017 23:09:48 +0000</created>
                <updated>Mon, 30 Oct 2023 23:14:08 +0000</updated>
                            <resolved>Tue, 12 Dec 2017 00:00:47 +0000</resolved>
                                    <version>3.4.6</version>
                    <version>3.6.0-rc4</version>
                                    <fixVersion>3.4.19</fixVersion>
                    <fixVersion>3.6.1</fixVersion>
                    <fixVersion>3.7.1</fixVersion>
                                    <component>Sharding</component>
                                        <votes>3</votes>
                                    <watches>19</watches>
                                                                                                                <comments>
                            <comment id="2097102" author="xgen-internal-githook" created="Thu, 20 Dec 2018 15:13:33 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;jsmulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;, &apos;name&apos;: &apos;Jack Mulrow&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30768&quot; title=&quot;Primary queries using maxTimeMS cause temporary shard write unavailability if ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30768&quot;&gt;&lt;del&gt;SERVER-30768&lt;/del&gt;&lt;/a&gt; Mark hosts as down on NetworkInterfaceExceededTimeLimit instead of ExceededTimeLimit&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 06b3488461d820c9081e5d7fca0fc028470922ea)&lt;br/&gt;
Branch: v3.4&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/9aafd59034c627b4a4cbda492bce3cc6cbef7b9a&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/9aafd59034c627b4a4cbda492bce3cc6cbef7b9a&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2096911" author="ramon.fernandez" created="Thu, 20 Dec 2018 11:35:28 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=paulw%40bitheads.com&quot; class=&quot;user-hover&quot; rel=&quot;paulw@bitheads.com&quot;&gt;paulw@bitheads.com&lt;/a&gt;, there&apos;s a &lt;a href=&quot;https://github.com/mongodb/mongo/pull/1280&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;pull request&lt;/a&gt; for the backport that is currently being assessed for quality and safety. We&apos;ll post updates to this ticket as they happen.&lt;/p&gt;

&lt;p&gt;Thanks for your continued patience,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="2084823" author="paulw@bitheads.com" created="Sat, 8 Dec 2018 18:23:43 +0000"  >&lt;p&gt;Question - is there a technical reason why this fix wasn&apos;t also applied to the v3.4 branch?&lt;/p&gt;

&lt;p&gt;We just upgraded from 3.2 -&amp;gt; 3.4, and our production servers are now hitting this issue... &#160;We&apos;re looking at the possibility of applying the same fix to the 3.4 branch and rebuilding - but maybe those more intimate with the code have a reason for why it wouldn&apos;t work? (and thus wasn&apos;t done?).&lt;/p&gt;

&lt;p&gt;Any thoughts would be appreciated. Thanks!&lt;/p&gt;

&lt;p&gt;Paul.&lt;/p&gt;</comment>
                            <comment id="1749183" author="xgen-internal-githook" created="Mon, 11 Dec 2017 23:52:39 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;, &apos;username&apos;: &apos;jsmulrow&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30768&quot; title=&quot;Primary queries using maxTimeMS cause temporary shard write unavailability if ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30768&quot;&gt;&lt;del&gt;SERVER-30768&lt;/del&gt;&lt;/a&gt; Mark hosts as down on NetworkInterfaceExceededTimeLimit instead of ExceededTimeLimit&lt;/p&gt;

&lt;p&gt;(cherry picked from commit 06b3488461d820c9081e5d7fca0fc028470922ea)&lt;br/&gt;
Branch: v3.6&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/5708ae71e0334874e63951a2868ec30e09287806&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/5708ae71e0334874e63951a2868ec30e09287806&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1748474" author="xgen-internal-githook" created="Mon, 11 Dec 2017 16:15:43 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Jack Mulrow&apos;, &apos;email&apos;: &apos;jack.mulrow@mongodb.com&apos;, &apos;username&apos;: &apos;jsmulrow&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-30768&quot; title=&quot;Primary queries using maxTimeMS cause temporary shard write unavailability if ExceededTimeLimit&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-30768&quot;&gt;&lt;del&gt;SERVER-30768&lt;/del&gt;&lt;/a&gt; Mark hosts as down on NetworkInterfaceExceededTimeLimit instead of ExceededTimeLimit&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/06b3488461d820c9081e5d7fca0fc028470922ea&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/06b3488461d820c9081e5d7fca0fc028470922ea&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1731968" author="kaloian.manassiev" created="Tue, 21 Nov 2017 19:01:08 +0000"  >&lt;p&gt;From looking at the code, I do not see a good reason to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/dcaf6432b7219d57cc5cf0de56b61264b8155e5d/src/mongo/s/client/shard_remote.cpp#L129&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;notify the replica set monitor&lt;/a&gt; on &lt;tt&gt;ExceededTimeLimit&lt;/tt&gt; errors, because these should be an indicator that a user-requested time limit was exceeded, but not necessarily a problem with the system.&lt;/p&gt;

&lt;p&gt;As part of this fix we should inspect all the places, which return &lt;tt&gt;ExceededTmeLimit&lt;/tt&gt; and also possibly add &lt;tt&gt;NetworkInterfaceExceededTimeLimit&lt;/tt&gt; to the list of errors, which mark shards as failed.&lt;/p&gt;</comment>
                            <comment id="1656399" author="oleg@evergage.com" created="Thu, 24 Aug 2017 17:30:38 +0000"  >&lt;p&gt;Thank you Mark!&lt;/p&gt;

&lt;p&gt;I&apos;ve been paying attention to the message for marking hosts as failed. I noticed that there&apos;s another circumstance that led to a primary being marked as unavailable:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2017-08-23T15:38:27.153+0000 I NETWORK  [conn28770] Marking host m17a.xyz:27017 as failed :: caused by :: RecvStaleConfig: can&apos;t query replica set node m17a.xyz:27017: stale config in DBClientCursor::dataReceived() ( ns : somedb.some_collection, received : 0|0||000000000000000000000000, wanted : 1|0||597ba0655167407959b03891, recv )&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Should this be a separate issue? It feels related to this. It seems incorrect to flag the entire primary as down just because there is a stale config data refresh happening for one collection (out of many).&lt;/p&gt;</comment>
                            <comment id="1654303" author="mark.agarunov" created="Tue, 22 Aug 2017 19:57:15 +0000"  >&lt;p&gt;Hello &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=oleg%40evergage.com&quot; class=&quot;user-hover&quot; rel=&quot;oleg@evergage.com&quot;&gt;oleg@evergage.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the detailed report. I&apos;ve managed to reproduce this behavior using the steps you outlined and have set the fixVersion to &quot;Needs Triage&quot; for this issue to be scheduled against our currently planned work. Updates will be posted on this ticket as they happen.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Mark&lt;/p&gt;</comment>
                            <comment id="1653381" author="oleg@evergage.com" created="Mon, 21 Aug 2017 23:18:27 +0000"  >&lt;p&gt;It looks like max_time_ms_sharded.js asserts only that the timeout was correctly sent back, but does not assert that the queried shard server is still reachable afterwards (and a primary is present).&lt;/p&gt;</comment>
                            <comment id="1653379" author="oleg@evergage.com" created="Mon, 21 Aug 2017 23:14:58 +0000"  >&lt;p&gt;I believe the problem lies within shard_remote.cpp in ShardRemote::updateReplSetMonitor:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.6/src/mongo/s/client/shard_remote.cpp#L135&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.4.6/src/mongo/s/client/shard_remote.cpp#L135&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;It considers ExceededTimeLimit to be an unreachable host situation. That might have been ok if maxTimeMS were used for internal system queries only. However, it&apos;s not OK when it&apos;s an application query specifying maxTimeMS.&lt;/p&gt;

&lt;p&gt;I recommend removing the clause that marks the host as failed on ExceededTimeLimit. That, or we have to detect whether the time limit was exceeded on a system or internal command or an application find.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10420">
                    <name>Backports</name>
                                            <outwardlinks description="backported by">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>10.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_12450" key="com.atlassian.jira.plugin.system.customfieldtypes:multicheckboxes">
                        <customfieldname>Backport Requested</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="15141"><![CDATA[v3.6]]></customfieldvalue>
    <customfieldvalue key="14340"><![CDATA[v3.4]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 22 Aug 2017 19:35:01 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 7 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 7 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jack.mulrow@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>mark.agarunov</customfieldvalue>
            <customfieldvalue>oleg@evergage.com</customfieldvalue>
            <customfieldvalue>paulw@bitheads.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htdcyn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hradq7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1980">Sharding 2017-12-04</customfieldvalue>
    <customfieldvalue id="2063">Sharding 2017-12-18</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;ol&gt;
	&lt;li&gt;Set up a sharded cluster with replica sets&lt;/li&gt;
	&lt;li&gt;Issue a find(query).maxTimeMS(1) command to cause a timeout, such that the Mongo shell prints:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;E QUERY    [thread1] Error: error: {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;ok&quot; : 0,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;errmsg&quot; : &quot;operation exceeded time limit&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;code&quot; : 50,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	&quot;codeName&quot; : &quot;ExceededTimeLimit&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;At this point, MongoS logs that it marked the primary as failed:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;# 2017-08-21T21:11:05.717+0000 I NETWORK  [NetworkInterfaceASIO-TaskExecutorPool-1-0] Marking host shard3.xyz:27017 as failed :: caused by :: ExceededTimeLimit: operation exceeded time limit&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;Immediately issue a write command, which will fail with:
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;error 133: Write failed with error code 133 and error message &apos;could not find host matching read preference { mode: &quot;primary&quot;, tags: [ {} ] } for set xyz_shard3&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
&lt;/ol&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htcz1j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>