<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:12:56 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-46968] Test locally that we eliminate retries of short-running operations due to planned mongos shutdown (without streamable isMaster)</title>
                <link>https://jira.mongodb.org/browse/SERVER-46968</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Write a workload of&#160;short-running reads and writes (&amp;lt;1 second). This should be run against a &lt;br/&gt;
 ShardingTest with 2 mongoses, where the two mongoses are alternately shut down. Run this workload with the 4.2 version of pymongo to test without streamable isMaster (consider using a smaller client heartbeat frequency or a longer quiesce mode, in order to be robust to slow machines).&#160;The test should disable retries in pymongo and assert that all commands succeed. This demonstrates that the need for retries is eliminated.&#160;Consider only running this test on variants that are not typically slow.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1280879">SERVER-46968</key>
            <summary>Test locally that we eliminate retries of short-running operations due to planned mongos shutdown (without streamable isMaster)</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="jason.chan@mongodb.com">Jason Chan</assignee>
                                    <reporter username="tess.avitabile@mongodb.com">Tess Avitabile</reporter>
                        <labels>
                    </labels>
                <created>Wed, 18 Mar 2020 20:03:08 +0000</created>
                <updated>Thu, 28 May 2020 20:19:27 +0000</updated>
                            <resolved>Thu, 28 May 2020 20:19:27 +0000</resolved>
                                                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>1</watches>
                                                                                                                <comments>
                            <comment id="3133061" author="jason.chan" created="Wed, 27 May 2020 23:44:52 +0000"  >&lt;p&gt;Similarly, we expect that if a mongos enters quiesce mode while a read is still in progress, the read should return successfully to the driver even if the driver has already marked the node as UNKNOWN.&lt;/p&gt;

&lt;p&gt;Our observations for this test case in a sharded cluster with two mongos nodes:&lt;/p&gt;

&lt;p&gt;We set two failpoints - one to hang a read on the mongos, and one to cause the same mongos to hang once it enters quiesce mode:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.056-0400 s20024| 2020-05-27T19:32:09.056-04:00 W  CONTROL  23829   [conn12] &quot;Set failpoint&quot;,&quot;attr&quot;:{&quot;failPointName&quot;:&quot;waitInFindBeforeMakingBatch&quot;,&quot;failPoint&quot;:{&quot;mode&quot;:1,&quot;data&quot;:{},&quot;timesEntered&quot;:0}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.057-0400 s20024| 2020-05-27T19:32:09.056-04:00 I  COMMAND  51803   [conn12] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;admin.waitInFindBeforeMakingBatch&quot;,&quot;appName&quot;:&quot;MongoDB Shell&quot;,&quot;command&quot;:{&quot;configureFailPoint&quot;:&quot;waitInFindBeforeMakingBatch&quot;,&quot;mode&quot;:&quot;alwaysOn&quot;,&quot;data&quot;:{},&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;f0d038a1-93c4-4185-8761-243cdb41b7cb&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;admin&quot;},&quot;numYields&quot;:0,&quot;reslen&quot;:174,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.262-0400 s20024| 2020-05-27T19:32:09.261-04:00 W  CONTROL  23829   [conn12] &quot;Set failpoint&quot;,&quot;attr&quot;:{&quot;failPointName&quot;:&quot;hangDuringQuiesceMode&quot;,&quot;failPoint&quot;:{&quot;mode&quot;:1,&quot;data&quot;:{},&quot;timesEntered&quot;:0}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.262-0400 s20024| 2020-05-27T19:32:09.261-04:00 I  COMMAND  51803   [conn12] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;admin.hangDuringQuiesceMode&quot;,&quot;appName&quot;:&quot;MongoDB Shell&quot;,&quot;command&quot;:{&quot;configureFailPoint&quot;:&quot;hangDuringQuiesceMode&quot;,&quot;mode&quot;:&quot;alwaysOn&quot;,&quot;data&quot;:{},&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;f0d038a1-93c4-4185-8761-243cdb41b7cb&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;admin&quot;},&quot;numYields&quot;:0,&quot;reslen&quot;:174,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Wait for the find to hang on mongos node s20024:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.225-0400 s20024| 2020-05-27T19:32:09.224-04:00 I  QUERY    20908   [conn17] &quot;Waiting in find before making batch for query - {cq_Short}&quot;,&quot;attr&quot;:{&quot;cq_Short&quot;:&quot;ns: test.state query: {} sort: {} projection: {} limit: 1&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Call shutdown on mongos node s20024 and wait for it to hang while in quiesce mode:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.267-0400 s20024| 2020-05-27T19:32:09.266-04:00 I  CONTROL  23377   [SignalHandler] &quot;got signal {sig} ({strsignal_sig})&quot;,&quot;attr&quot;:{&quot;sig&quot;:15,&quot;strsignal_sig&quot;:&quot;Terminated&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.267-0400 s20024| 2020-05-27T19:32:09.266-04:00 I  CONTROL  23378   [SignalHandler] &quot;kill from pid:{si_si_pid} uid:{si_si_uid}&quot;,&quot;attr&quot;:{&quot;si_si_pid&quot;:9732,&quot;si_si_uid&quot;:1000}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.267-0400 s20024| 2020-05-27T19:32:09.266-04:00 I  CONTROL  23381   [SignalHandler] &quot;will terminate after current cmd ends&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.267-0400 s20024| 2020-05-27T19:32:09.266-04:00 I  COMMAND  4695700 [SignalHandler] &quot;hangDuringQuiesceMode failpoint enabled&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:09.267-0400 s20024| 2020-05-27T19:32:09.267-04:00 I  COMMAND  51803   [conn12] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;admin.&quot;,&quot;appName&quot;:&quot;MongoDB Shell&quot;,&quot;command&quot;:{&quot;waitForFailPoint&quot;:&quot;hangDuringQuiesceMode&quot;,&quot;timesEntered&quot;:1,&quot;maxTimeMS&quot;:300000,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;f0d038a1-93c4-4185-8761-243cdb41b7cb&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;admin&quot;},&quot;numYields&quot;:0,&quot;reslen&quot;:163,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Wait for mongos to reply to the driver with a ShutdownInProgress isMaster response:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:10.852-0400 s20024| 2020-05-27T19:32:10.852-04:00 D1 -        23074   [conn16] &quot;User assertion&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;ShutdownInProgress: Mongos is in quiesce mode and will shut down&quot;,&quot;file&quot;:&quot;src/mongo/s/mongos_topology_coordinator.cpp&quot;,&quot;line&quot;:117}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:10.853-0400 s20024| 2020-05-27T19:32:10.852-04:00 D1 SHARDING 22772   [conn16] &quot;Exception thrown while processing command&quot;,&quot;attr&quot;:{&quot;db&quot;:&quot;admin&quot;,&quot;headerId&quot;:1350490027,&quot;error&quot;:&quot;ShutdownInProgress: Mongos is in quiesce mode and will shut down&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:10.853-0400 s20024| 2020-05-27T19:32:10.852-04:00 I  COMMAND  51803   [conn16] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;admin.$cmd&quot;,&quot;command&quot;:{&quot;ismaster&quot;:1,&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;admin&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;numYields&quot;:0,&quot;ok&quot;:0,&quot;errMsg&quot;:&quot;Mongos is in quiesce mode and will shut down&quot;,&quot;errName&quot;:91,&quot;errCode&quot;:91,&quot;reslen&quot;:325,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We disable the failpoint to allow the hanging read to finish:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:14.278-0400 s20024| 2020-05-27T19:32:14.278-04:00 I  COMMAND  51803   [conn17] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;find&quot;:&quot;state&quot;,&quot;filter&quot;:{},&quot;limit&quot;:1,&quot;singleBatch&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;d73ff390-6f73-44fe-bb49-daf53853fc18&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:0,&quot;nreturned&quot;:1,&quot;reslen&quot;:272,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:5064}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Future reads are sent to the other mongos node s20025. The logs do not show node s20024 receiving anymore reads for the rest of quiesce mode:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster_hanging_read] 2020-05-27T19:32:14.532-0400 s20025| 2020-05-27T19:32:14.531-04:00 I  COMMAND  51803   [conn15] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;find&quot;:&quot;state&quot;,&quot;filter&quot;:{},&quot;limit&quot;:1,&quot;singleBatch&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;d73ff390-6f73-44fe-bb49-daf53853fc18&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590622328,&quot;i&quot;:7}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:0,&quot;nreturned&quot;:1,&quot;reslen&quot;:269,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The test completes successfully with driver option retryReads set to false, indicating that all reads completed successfully without the driver ever having to retry.&lt;/p&gt;

&lt;p&gt;Example logs for this test case:   &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/262986/262986_mongos_quiesce_mode_non_streamable_ismaster_with_hanging_read.txt&quot; title=&quot;mongos_quiesce_mode_non_streamable_ismaster_with_hanging_read.txt attached to SERVER-46968&quot;&gt;mongos_quiesce_mode_non_streamable_ismaster_with_hanging_read.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                            <comment id="3133017" author="jason.chan" created="Wed, 27 May 2020 23:14:01 +0000"  >&lt;p&gt;For this test, we expect to observe that as a mongos enters quiesce mode, it should still be able to service short-running reads and writes. Once the mongos returns a ShutdownInProgress error to the driver, the driver should know to no longer target the shutting down mongos for future operations.&lt;/p&gt;

&lt;p&gt;Our observations for this test case in a sharded cluster with two mongos nodes:&lt;/p&gt;

&lt;p&gt;mongos node s20025 receives a series of read and write operations:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.596-0400 s20025| 2020-05-28T00:15:26.595-04:00 I  COMMAND  51803   [conn16] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;update&quot;:&quot;state&quot;,&quot;ordered&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639326,&quot;i&quot;:2}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;nMatched&quot;:1,&quot;nModified&quot;:1,&quot;numYields&quot;:0,&quot;reslen&quot;:185,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;nMatched&quot;:1,&quot;nModified&quot;:1,&quot;numYields&quot;:0,&quot;reslen&quot;:185,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.597-0400 s20025| 2020-05-28T00:15:26.597-04:00 I  COMMAND  51803   [conn16] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;find&quot;:&quot;state&quot;,&quot;filter&quot;:{},&quot;limit&quot;:1,&quot;singleBatch&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639326,&quot;i&quot;:3}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:0,&quot;nreturned&quot;:1,&quot;reslen&quot;:279,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;mongos node s20025 receives a shutdown command and enters quiesce mode:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.791-0400 s20025| 2020-05-28T00:15:26.790-04:00 I  CONTROL  23377   [SignalHandler] &quot;got signal {sig} ({strsignal_sig})&quot;,&quot;attr&quot;:{&quot;sig&quot;:15,&quot;strsignal_sig&quot;:&quot;Terminated&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.791-0400 s20025| 2020-05-28T00:15:26.790-04:00 I  CONTROL  23378   [SignalHandler] &quot;kill from pid:{si_si_pid} uid:{si_si_uid}&quot;,&quot;attr&quot;:{&quot;si_si_pid&quot;:18828,&quot;si_si_uid&quot;:1000}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.791-0400 s20025| 2020-05-28T00:15:26.790-04:00 I  CONTROL  23381   [SignalHandler] &quot;will terminate after current cmd ends&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.791-0400 s20025| 2020-05-28T00:15:26.791-04:00 I  COMMAND  4695701 [SignalHandler] &quot;Entering quiesce mode for mongos shutdown&quot;,&quot;attr&quot;:{&quot;quiesceTimeMillis&quot;:5000}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;mongos node s20025 is still able to service operations while in quiesce mode:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.851-0400 s20025| 2020-05-28T00:15:26.850-04:00 I  COMMAND  51803   [conn16] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;update&quot;:&quot;state&quot;,&quot;ordered&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639326,&quot;i&quot;:3}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;nMatched&quot;:1,&quot;nModified&quot;:1,&quot;numYields&quot;:0,&quot;reslen&quot;:185,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&#8230;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:26.853-0400 s20025| 2020-05-28T00:15:26.853-04:00 I  COMMAND  51803   [conn16] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;find&quot;:&quot;state&quot;,&quot;filter&quot;:{},&quot;limit&quot;:1,&quot;singleBatch&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639326,&quot;i&quot;:4}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:0,&quot;nreturned&quot;:1,&quot;reslen&quot;:279,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;mongos node s20025 eventually replies to the driver with a ShutdownInProgress isMaster response:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.367-0400 s20025| 2020-05-28T00:15:27.367-04:00 I  NETWORK  51800   [conn21] &quot;client metadata&quot;,&quot;attr&quot;:{&quot;remote&quot;:&quot;127.0.0.1:44210&quot;,&quot;client&quot;:&quot;conn21&quot;,&quot;doc&quot;:{&quot;driver&quot;:{&quot;name&quot;:&quot;PyMongo&quot;,&quot;version&quot;:&quot;3.10.1&quot;},&quot;os&quot;:{&quot;type&quot;:&quot;Linux&quot;,&quot;name&quot;:&quot;Linux&quot;,&quot;architecture&quot;:&quot;x86_64&quot;,&quot;version&quot;:&quot;4.13.0-46-generic&quot;},&quot;platform&quot;:&quot;CPython 3.7.0.final.0&quot;}}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.367-0400 s20025| 2020-05-28T00:15:27.367-04:00 D1 -        23074   [conn21] &quot;User assertion&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;ShutdownInProgress: Mongos is in quiesce mode and will shut down&quot;,&quot;file&quot;:&quot;src/mongo/s/mongos_topology_coordinator.cpp&quot;,&quot;line&quot;:117}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.368-0400 s20025| 2020-05-28T00:15:27.367-04:00 D1 SHARDING 22772   [conn21] &quot;Exception thrown while processing command&quot;,&quot;attr&quot;:{&quot;db&quot;:&quot;admin&quot;,&quot;headerId&quot;:1141616124,&quot;error&quot;:&quot;ShutdownInProgress: Mongos is in quiesce mode and will shut down&quot;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.368-0400 s20025| 2020-05-28T00:15:27.367-04:00 I  COMMAND  51803   [conn21] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;admin.$cmd&quot;,&quot;command&quot;:{&quot;ismaster&quot;:1,&quot;client&quot;:{&quot;driver&quot;:{&quot;name&quot;:&quot;PyMongo&quot;,&quot;version&quot;:&quot;3.10.1&quot;},&quot;os&quot;:{&quot;type&quot;:&quot;Linux&quot;,&quot;name&quot;:&quot;Linux&quot;,&quot;architecture&quot;:&quot;x86_64&quot;,&quot;version&quot;:&quot;4.13.0-46-generic&quot;},&quot;platform&quot;:&quot;CPython 3.7.0.final.0&quot;},&quot;$db&quot;:&quot;admin&quot;},&quot;numYields&quot;:0,&quot;ok&quot;:0,&quot;errMsg&quot;:&quot;Mongos is in quiesce mode and will shut down&quot;,&quot;errName&quot;:91,&quot;errCode&quot;:91,&quot;reslen&quot;:340,&quot;protocol&quot;:&quot;op_query&quot;,&quot;durationMillis&quot;:0}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We see that all future operations are sent to the other mongos node s20024:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.623-0400 s20024| 2020-05-28T00:15:27.622-04:00 I  COMMAND  51803   [conn17] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;update&quot;:&quot;state&quot;,&quot;ordered&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639327,&quot;i&quot;:2}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;nMatched&quot;:1,&quot;nModified&quot;:1,&quot;numYields&quot;:0,&quot;reslen&quot;:185,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:4}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:mongos_quiesce_mode_non_streamable_ismaster] 2020-05-28T00:15:27.625-0400 s20024| 2020-05-28T00:15:27.625-04:00 I  COMMAND  51803   [conn17] &quot;Slow query&quot;,&quot;attr&quot;:{&quot;type&quot;:&quot;command&quot;,&quot;ns&quot;:&quot;test.state&quot;,&quot;command&quot;:{&quot;find&quot;:&quot;state&quot;,&quot;filter&quot;:{},&quot;limit&quot;:1,&quot;singleBatch&quot;:true,&quot;lsid&quot;:{&quot;id&quot;:{&quot;$uuid&quot;:&quot;fa1fa44a-1501-4e63-afe5-ed285e530c98&quot;}},&quot;$clusterTime&quot;:{&quot;clusterTime&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1590639327,&quot;i&quot;:3}},&quot;signature&quot;:{&quot;hash&quot;:{&quot;$binary&quot;:{&quot;base64&quot;:&quot;AAAAAAAAAAAAAAAAAAAAAAAAAAA=&quot;,&quot;subType&quot;:&quot;0&quot;}},&quot;keyId&quot;:0}},&quot;$db&quot;:&quot;test&quot;,&quot;$readPreference&quot;:{&quot;mode&quot;:&quot;primary&quot;}},&quot;nShards&quot;:1,&quot;cursorExhausted&quot;:true,&quot;numYields&quot;:0,&quot;nreturned&quot;:1,&quot;reslen&quot;:279,&quot;protocol&quot;:&quot;op_msg&quot;,&quot;durationMillis&quot;:1}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;The test completes successfully with both driver options retryReads and retryWrites set to false, indicating that all operations completed successfully without the driver ever having to retry.&lt;/p&gt;

&lt;p&gt;Example logs for this test case:  &lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/263006/263006_mongos_quiesce_mode_non_streamable_ismaster.txt&quot; title=&quot;mongos_quiesce_mode_non_streamable_ismaster.txt attached to SERVER-46968&quot;&gt;mongos_quiesce_mode_non_streamable_ismaster.txt&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                            <outwardlinks description="depends on">
                                        <issuelink>
            <issuekey id="1280641">SERVER-46958</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1280846">SERVER-46963</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1283990">SERVER-47018</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="1280884">SERVER-46969</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="263006" name="mongos_quiesce_mode_non_streamable_ismaster.txt" size="1944841" author="jason.chan@mongodb.com" created="Thu, 28 May 2020 04:22:36 +0000"/>
                            <attachment id="262986" name="mongos_quiesce_mode_non_streamable_ismaster_with_hanging_read.txt" size="1406587" author="jason.chan@mongodb.com" created="Wed, 27 May 2020 23:49:06 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 27 May 2020 23:14:01 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 37 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<s><a href='https://jira.mongodb.org/browse/SERVER-46958'>SERVER-46958</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-46963'>SERVER-46963</a></s>, <s><a href='https://jira.mongodb.org/browse/SERVER-47018'>SERVER-47018</a></s>]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-1703</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 37 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>tess.avitabile@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx9n0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hwxecv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3934">Repl 2020-06-01</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hx99a7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>