<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:09:21 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-25497] Fix sharded query path to handle shutdown of the mongos process</title>
                <link>https://jira.mongodb.org/browse/SERVER-25497</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The &lt;tt&gt;AsyncResultsMerger&lt;/tt&gt; is the component of the new mongos find execution path introduced in version 3.2.0. It uses the &lt;tt&gt;TaskExecutor&lt;/tt&gt; to run &lt;tt&gt;find&lt;/tt&gt; and &lt;tt&gt;getMore&lt;/tt&gt; commands against the shards. During shutdown of the mongos process, the &lt;tt&gt;TaskExecutor&lt;/tt&gt; shutdown sequence occurs, during which mongos cursors may still exist but the &lt;tt&gt;TaskExecutor&lt;/tt&gt; can no longer be used to run commands on remote nodes.&lt;/p&gt;

&lt;p&gt;The &lt;tt&gt;AsyncResultsMerger&lt;/tt&gt; does not behave correctly during &lt;tt&gt;TaskExecutor&lt;/tt&gt; shutdown. This can cause mongos to crash during shutdown in one of several ways.&lt;/p&gt;

&lt;h5&gt;&lt;a name=&quot;Originaldescription&quot;&gt;&lt;/a&gt;Original description&lt;/h5&gt;

&lt;p&gt;Hi all,&lt;/p&gt;

&lt;p&gt;I found the following stack trace and I am not sure what is the root cause of this. It happened on all the mongos with a few minutes interval. It did occur after a signal 15 so I do not know if that is the reason.&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-08-05T08:03:08.269Z I -        [conn798228] Invariant failure event.isValid() src/mongo/executor/thread_pool_task_executor.cpp 234&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-08-05T08:03:08.269Z I -        [conn798228] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after invariant() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2016-08-05T08:03:08.271Z F -        [conn799245] Got signal: 6 (Aborted).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; 0xc5dca2 0xc5cbc9 0xc5d3d2 0x7f03252f2cb0 0x7f0324f5b035 0x7f0324f5e79b 0xbe245b 0xa13cb8 0xb8a68c 0xb81536 0xb8777a 0xba0ee7 0xb8e8fb 0x672115 0xc07715 0x7f03252eae9a 0x7f032501836d&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;----- BEGIN BACKTRACE -----&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{&quot;backtrace&quot;:[{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;85DCA2&quot;,&quot;s&quot;:&quot;_ZN5mongo15printStackTraceERSo&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;85CBC9&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;85D3D2&quot;},{&quot;b&quot;:&quot;7F03252E3000&quot;,&quot;o&quot;:&quot;FCB0&quot;},{&quot;b&quot;:&quot;7F0324F25000&quot;,&quot;o&quot;:&quot;36035&quot;,&quot;s&quot;:&quot;gsignal&quot;},{&quot;b&quot;:&quot;7F0324F25000&quot;,&quot;o&quot;:&quot;3979B&quot;,&quot;s&quot;:&quot;abort&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;7E245B&quot;,&quot;s&quot;:&quot;_ZN5mongo15invariantFailedEPKcS1_j&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;613CB8&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;78A68C&quot;,&quot;s&quot;:&quot;_ZN5mongo16RouterStageMerge4killEv&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;781536&quot;,&quot;s&quot;:&quot;_ZN5mongo24ClusterClientCursorGuardD1Ev&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;78777A&quot;,&quot;s&quot;:&quot;_ZN5mongo11ClusterFind8runQueryEPNS_16OperationContextERKNS_14CanonicalQueryERKNS_21ReadPreferenceSettingEPSt6vectorINS_7BSONObjESaISA_EE&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;7A0EE7&quot;,&quot;s&quot;:&quot;_ZN5mongo8Strategy7queryOpEPNS_16OperationContextERNS_7RequestE&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;78E8FB&quot;,&quot;s&quot;:&quot;_ZN5mongo7Request7processEPNS_16OperationContextEi&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;272115&quot;,&quot;s&quot;:&quot;_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE&quot;},{&quot;b&quot;:&quot;400000&quot;,&quot;o&quot;:&quot;807715&quot;,&quot;s&quot;:&quot;_ZN5mongo17PortMessageServer17handleIncomingMsgEPv&quot;},{&quot;b&quot;:&quot;7F03252E3000&quot;,&quot;o&quot;:&quot;7E9A&quot;},{&quot;b&quot;:&quot;7F0324F25000&quot;,&quot;o&quot;:&quot;F336D&quot;,&quot;s&quot;:&quot;clone&quot;}],&quot;processInfo&quot;:{ &quot;mongodbVersion&quot; : &quot;3.2.8&quot;, &quot;gitVersion&quot; : &quot;ed70e33130c977bda0024c125b56d159573dbaf0&quot;, &quot;compiledModules&quot; : [], &quot;uname&quot; : { &quot;sysname&quot; : &quot;Linux&quot;, &quot;release&quot; : &quot;3.2.0-77-virtual&quot;, &quot;version&quot; : &quot;#114-Ubuntu SMP Tue Mar 10 17:38:02 UTC 2015&quot;, &quot;machine&quot; : &quot;x86_64&quot; }, &quot;somap&quot; : [ { &quot;elfType&quot; : 2, &quot;b&quot; : &quot;400000&quot;, &quot;buildId&quot; : &quot;16C848A3171CB2CF2FDC800A602C9F785595A08F&quot; }, { &quot;b&quot; : &quot;7FFF0700B000&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;4CF95530ACFE32D062E1E8F7524D5E956AF54586&quot; }, { &quot;b&quot; : &quot;7F03261FA000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libssl.so.1.0.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;05BB9627FE4CCFC087FC0A1E064098BAAAF0ABE2&quot; }, { &quot;b&quot; : &quot;7F0325E1E000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libcrypto.so.1.0.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;88FFD2D95DF12BF9CA931AF6E50F42D83A98C3FE&quot; }, { &quot;b&quot; : &quot;7F0325C16000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/librt.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;8C19981216B96C14A1C82A4C884D1FA7FECB7979&quot; }, { &quot;b&quot; : &quot;7F0325A12000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libdl.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;68048F961CC6B636BC27B40DE0DD8B8632B10180&quot; }, { &quot;b&quot; : &quot;7F0325716000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libm.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;97900BE41183B1AB4E6322A8FE08D68315FED60A&quot; }, { &quot;b&quot; : &quot;7F0325500000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libgcc_s.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;ECF322A96E26633C5D10F18215170DD4395AF82C&quot; }, { &quot;b&quot; : &quot;7F03252E3000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libpthread.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;9E17F98B86F7723CA1DC42235AEAB33B86FCB89D&quot; }, { &quot;b&quot; : &quot;7F0324F25000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libc.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;FC161CFC250D850EB08A7B76BAD5D2022AAEBB10&quot; }, { &quot;b&quot; : &quot;7F0326458000&quot;, &quot;path&quot; : &quot;/lib64/ld-linux-x86-64.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;B60D4C2FCB331CEC2764B936F83C0622676CF9ED&quot; }, { &quot;b&quot; : &quot;7F0324D0E000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libz.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;F695ECFCF3918D5D34989398A14B7ECDD9F46CD0&quot; } ] }}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo15printStackTraceERSo+0x32) [0xc5dca2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(+0x85CBC9) [0xc5cbc9]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(+0x85D3D2) [0xc5d3d2]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; libpthread.so.0(+0xFCB0) [0x7f03252f2cb0]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; libc.so.6(gsignal+0x35) [0x7f0324f5b035]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; libc.so.6(abort+0x17B) [0x7f0324f5e79b]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xCB) [0xbe245b]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(+0x613CB8) [0xa13cb8]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo16RouterStageMerge4killEv+0x2C) [0xb8a68c]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo24ClusterClientCursorGuardD1Ev+0x46) [0xb81536]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo11ClusterFind8runQueryEPNS_16OperationContextERKNS_14CanonicalQueryERKNS_21ReadPreferenceSettingEPSt6vectorINS_7BSONObjESaISA_EE+0xCFA) [0xb8777a]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo8Strategy7queryOpEPNS_16OperationContextERNS_7RequestE+0x567) [0xba0ee7]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo7Request7processEPNS_16OperationContextEi+0x5CB) [0xb8e8fb]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x65) [0x672115]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0xc07715]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; libpthread.so.0(+0x7E9A) [0x7f03252eae9a]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; libc.so.6(clone+0x6D) [0x7f032501836d]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;-----  END BACKTRACE  -----&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</description>
                <environment></environment>
        <key id="307412">SERVER-25497</key>
            <summary>Fix sharded query path to handle shutdown of the mongos process</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="ruoxin.xu@mongodb.com">Ruoxin Xu</assignee>
                                    <reporter username="davenson.lombard@mongodb.com">Davenson Lombard</reporter>
                        <labels>
                            <label>mongos-drain-mode-fallout</label>
                            <label>qexec-team</label>
                            <label>query-44-grooming</label>
                    </labels>
                <created>Mon, 8 Aug 2016 21:54:21 +0000</created>
                <updated>Fri, 16 Oct 2020 08:27:07 +0000</updated>
                            <resolved>Thu, 15 Oct 2020 14:24:05 +0000</resolved>
                                    <version>3.4.0-rc0</version>
                                    <fixVersion>4.9.0</fixVersion>
                                    <component>Sharding</component>
                                        <votes>0</votes>
                                    <watches>22</watches>
                                                                                                                <comments>
                            <comment id="3445915" author="xgen-internal-githook" created="Thu, 15 Oct 2020 12:45:37 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Ruoxin Xu&apos;, &apos;email&apos;: &apos;ruoxin.xu@mongodb.com&apos;, &apos;username&apos;: &apos;RuoxinXu&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-25497&quot; title=&quot;Fix sharded query path to handle shutdown of the mongos process&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-25497&quot;&gt;&lt;del&gt;SERVER-25497&lt;/del&gt;&lt;/a&gt; Fix sharded query path to handle shutdown of the mongos process&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/4530bcb094d5017599699a3da074a061a493f2b1&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/4530bcb094d5017599699a3da074a061a493f2b1&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1817563" author="ian.boros" created="Tue, 27 Feb 2018 19:08:47 +0000"  >&lt;p&gt;Also! Note that there&apos;s another subtle bug with how the ARM deals with shutdown: When it &lt;a href=&quot;https://github.com/mongodb/mongo/blob/e3f361769cd13ba88aa24c1c0a71c76b187f64dd/src/mongo/s/query/async_results_merger.cpp#L674&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;checks for shutdown and returns an invalid event&lt;/a&gt;, the caller will likely free the ARM immediately. This is not safe because there may be outstanding callbacks, which, when run will try to access the ARM.&lt;/p&gt;

&lt;p&gt;Both of the solutions we came up with above (changing the shutdown order and using futures) should solve this problem as well.&lt;/p&gt;</comment>
                            <comment id="1817528" author="ian.boros" created="Tue, 27 Feb 2018 18:49:53 +0000"  >&lt;p&gt;We looked into fixing this as part of the sharded kill project.&lt;/p&gt;

&lt;p&gt;The fact that an event is double-signaled is a larger problem that&apos;s outside the scope of this ticket. I&apos;ve filed a ticket about this particular problem here: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33522&quot; title=&quot;Possible to call TaskExecutor::signalEvent twice during shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33522&quot;&gt;&lt;del&gt;SERVER-33522&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;As far as fixing the problem with the ARM:&lt;/p&gt;

&lt;p&gt;After more discussion with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.storch&quot; class=&quot;user-hover&quot; rel=&quot;david.storch&quot;&gt;david.storch&lt;/a&gt;, we&apos;ve agreed that doing what he suggests in the comment above could be too costly to performance on point-finds. We could add partition-locking to the &lt;tt&gt;ClusterCursorManager&lt;/tt&gt; to mitigate this, but it could still affect performance.&lt;/p&gt;

&lt;p&gt;Another way to fix this problem would be to change the order of shutdown so that the TaskExecutor shuts down after all client threads have terminated. However, &lt;a href=&quot;https://github.com/mongodb/mongo/blob/4a69dcaf78f29cf440ada961c7e59de6b5dde111/src/mongo/s/server.cpp#L247-L253&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;we only join on client threads when running under ASAN, and even then, we do so with a timeout&lt;/a&gt;. Changing the mongos to join with all client threads would probably be a significant amount of work.&lt;/p&gt;

&lt;p&gt;Yet another alternative would be to use &quot;Events&quot; which are separate from the TaskExecutor, and still work even when shutdown is happening. After talking with &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=redbeard0531&quot; class=&quot;user-hover&quot; rel=&quot;redbeard0531&quot;&gt;redbeard0531&lt;/a&gt; in-person, it seems like &quot;Futures&quot; will provide exactly this. Therefore I think that modifying the ARM to use Futures rather than TaskExecutor Events should fix the problems around the ARM and shutdown. Since this work of &quot;futurizing&quot; the ARM is already planned, I think we should just wait until we have futures to work on this.&lt;/p&gt;</comment>
                            <comment id="1461441" author="david.storch" created="Mon, 19 Dec 2016 19:56:31 +0000"  >&lt;p&gt;After further investigation by the MongoDB Query Team, we have decided to mitigate the problem by fixing the initially reported crash under related ticket &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27460&quot; title=&quot;RouterStageMerge::kill() can crash during mongos shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27460&quot;&gt;&lt;del&gt;SERVER-27460&lt;/del&gt;&lt;/a&gt;. Please watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27460&quot; title=&quot;RouterStageMerge::kill() can crash during mongos shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27460&quot;&gt;&lt;del&gt;SERVER-27460&lt;/del&gt;&lt;/a&gt; for updates on this partial fix.&lt;/p&gt;

&lt;p&gt;In order to fix the problem in a complete way, we would have to make the following changes:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Prevent new mongos cursors from coming into being while the process is shutting down.&lt;/li&gt;
	&lt;li&gt;Wait for all existing cursors to be cleaned up before proceeding with the rest of shutdown, including shutting down the &lt;tt&gt;TaskExecutor&lt;/tt&gt;.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;We already shut down the mongos cursor manager before shutting down the TaskExecutor and we already prevent registration of new cursors once the &lt;tt&gt;ClusterCursorManager&lt;/tt&gt; begins to shut down. Therefore, in order to acheive #1, all we need to do is ensure that mongos cursors come into existence registered. This would prevent the creation of any new cursors once &lt;tt&gt;ClusterCursorManager::shutdown()&lt;/tt&gt; begins. It is an easy code change to make, but could come with a performance cost. In fact, the implementation used to work this way, but was changed in &lt;a href=&quot;https://github.com/mongodb/mongo/commit/5edafdbf6ca1effcb18d62c8e53b37544afecfcc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this commit&lt;/a&gt; in order to improve performance.&lt;/p&gt;

&lt;p&gt;One way to achieve #2 would be to mark all outstanding cursors as kill pending and wait for them to die before proceeding with shutdown. This is an unacceptable approach because it could delay shutdown for large amounts of time while the shards complete query execution work. Therefore, we need a way to preemptively kill an active mongos cursor by terminating any cursors which are in use on the shards. Allowing pinned cursors to be killed, &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21710&quot; title=&quot;Allow pinned ClientCursors to be killed on mongod&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21710&quot;&gt;&lt;del&gt;SERVER-21710&lt;/del&gt;&lt;/a&gt;, is a prerequisite for this work. Building a cross-shard preemptive mongos cursor-killing mechanism for shutdown would bring us a long way towards &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-6496&quot; title=&quot;provide a way to kill a sharded query on all shards&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-6496&quot;&gt;&lt;del&gt;SERVER-6496&lt;/del&gt;&lt;/a&gt;, the ability to quickly terminate a sharded query on all participating nodes.&lt;/p&gt;

&lt;p&gt;The complexity of these changes makes it unlikely that we will deliver the complete fix in 3.2 or 3.4, which is why we have chosen to work on the smaller, mitigating fix in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-27460&quot; title=&quot;RouterStageMerge::kill() can crash during mongos shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-27460&quot;&gt;&lt;del&gt;SERVER-27460&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;CC &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1420582" author="david.storch" created="Fri, 28 Oct 2016 15:37:11 +0000"  >&lt;p&gt;Our testing has revealed a similar race which can cause mongos to crash during shutdown with the following invariant failure:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.637+0000 s20765| 2016-08-28T18:23:49.636+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765| 2016-08-28T18:23:49.639+0000 I -        [ClusterCursorCleanupJob] Invariant failure !eventState-&amp;gt;isSignaledFlag src/mongo/executor/thread_pool_task_executor.cpp 433&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765| 2016-08-28T18:23:49.639+0000 I -        [ClusterCursorCleanupJob]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765|&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765| ***aborting after invariant() failure&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765|&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.640+0000 s20765|&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.643+0000 s20765| 2016-08-28T18:23:49.643+0000 F -        [ClusterCursorCleanupJob] Got signal: 6 (Aborted).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.643+0000 s20765|&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.643+0000 s20765|  0x55e8678048d1 0x55e86905b0b3 0x55e869059ca3 0x7fee30a383d0 0x7fee3047d418 0x7fee3047f01a 0x55e868e68bf1 0x55e8687cd021 0x55e8687d088e 0x55e868d0e07b 0x55e868d3f724 0x55e868d1d7ed 0x55e868d19a54 0x55e868e789ce 0x7fee30cfcc80 0x7fee30a2e6fa 0x7fee3054eb5d&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.644+0000 s20765| ----- BEGIN BACKTRACE -----&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765| {&quot;backtrace&quot;:[{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;8F08D1&quot;,&quot;s&quot;:&quot;__interceptor_backtrace&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;21470B3&quot;,&quot;s&quot;:&quot;_ZN5mongo15printStackTraceERSo&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;2145CA3&quot;},{&quot;b&quot;:&quot;7FEE30A27000&quot;,&quot;o&quot;:&quot;113D0&quot;},{&quot;b&quot;:&quot;7FEE30448000&quot;,&quot;o&quot;:&quot;35418&quot;,&quot;s&quot;:&quot;gsignal&quot;},{&quot;b&quot;:&quot;7FEE30448000&quot;,&quot;o&quot;:&quot;3701A&quot;,&quot;s&quot;:&quot;abort&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1F54BF1&quot;,&quot;s&quot;:&quot;_ZN5mongo15invariantFailedEPKcS1_j&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;18B9021&quot;,&quot;s&quot;:&quot;_ZN5mongo8executor22ThreadPoolTaskExecutor18signalEvent_inlockERKNS0_12TaskExecutor11EventHandleESt11unique_lockISt5mutexE&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;18BC88E&quot;,&quot;s&quot;:&quot;_ZN5mongo8executor22ThreadPoolTaskExecutor11signalEventERKNS0_12TaskExecutor11EventHandleE&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1DFA07B&quot;,&quot;s&quot;:&quot;_ZN5mongo18AsyncResultsMerger4killEv&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1E2B724&quot;,&quot;s&quot;:&quot;_ZN5mongo16RouterStageMerge4killEv&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1E097ED&quot;,&quot;s&quot;:&quot;_ZN5mongo20ClusterCursorManager17reapZombieCursorsEv&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1E05A54&quot;,&quot;s&quot;:&quot;_ZN5mongo23ClusterCursorCleanupJob3runEv&quot;},{&quot;b&quot;:&quot;55E866F14000&quot;,&quot;o&quot;:&quot;1F649CE&quot;,&quot;s&quot;:&quot;_ZN5mongo13BackgroundJob7jobBodyEv&quot;},{&quot;b&quot;:&quot;7FEE30C44000&quot;,&quot;o&quot;:&quot;B8C80&quot;},{&quot;b&quot;:&quot;7FEE30A27000&quot;,&quot;o&quot;:&quot;76FA&quot;},{&quot;b&quot;:&quot;7FEE30448000&quot;,&quot;o&quot;:&quot;106B5D&quot;,&quot;s&quot;:&quot;clone&quot;}],&quot;processInfo&quot;:{ &quot;mongodbVersion&quot; : &quot;3.3.11-240-g38e2879&quot;, &quot;gitVersion&quot; : &quot;38e287962414f62323abb93a67c5561a0d3175b3&quot;, &quot;compiledModules&quot; : [ &quot;enterprise&quot; ], &quot;uname&quot; : { &quot;sysname&quot; : &quot;Linux&quot;, &quot;release&quot; : &quot;4.4.0-21-generic&quot;, &quot;version&quot; : &quot;#37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016&quot;, &quot;machine&quot; : &quot;x86_64&quot; }, &quot;somap&quot; : [ { &quot;b&quot; : &quot;55E866F14000&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;AF5809D4F4DD2E8FDE8B073A4CA74AF0BC677ACB&quot; }, { &quot;b&quot; : &quot;7FFE3EFFA000&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;19C37034D41570D4CBC5A74A3874BC0C32CBCA5B&quot; }, { &quot;b&quot; : &quot;7FEE32413000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;ADF6F41F223D42193165FA0C55871F02D915FB19&quot; }, { &quot;b&quot; : &quot;7FEE32204000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;6B9F4061A1D44813A54DA4DBB0088F529D8D78EA&quot; }, { &quot;b&quot; : &quot;7FEE31FE9000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libsasl2.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;96BCC7EB28D81B1469EED6F24FC083CBD58577BC&quot; }, { &quot;b&quot; : &quot;7FEE31D9F000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;54EB3E674DB58B5BAEF009CBC4AB02CCF5386CA7&quot; }, { &quot;b&quot; : &quot;7FEE31A96000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libm.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;5C4078C04888A418F3DB0868702ECFDB35B3AD8B&quot; }, { &quot;b&quot; : &quot;7FEE3182D000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libssl.so.1.0.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;CC601E6C47AE87C33EE639D61CE08220A98A263F&quot; }, { &quot;b&quot; : &quot;7FEE313D2000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libcrypto.so.1.0.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;4BE6A02E9E96DDC6E8D22F73749388923AE5652B&quot; }, { &quot;b&quot; : &quot;7FEE311CA000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/librt.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;A779DBCB3A477DC0C8D09B60FAC7335D396C19DF&quot; }, { &quot;b&quot; : &quot;7FEE30FC6000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libdl.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;A2ADF3615338D49C702C41EB83A99AB743D2B574&quot; }, { &quot;b&quot; : &quot;7FEE30C44000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libstdc++.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;FCBB11A30679DE016410A0CDB6CCF0C2D11FD093&quot; }, { &quot;b&quot; : &quot;7FEE30A27000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libpthread.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;B77847CC9CACBCA3B5753D0D25A32E5795AFE75B&quot; }, { &quot;b&quot; : &quot;7FEE30811000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libgcc_s.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434&quot; }, { &quot;b&quot; : &quot;7FEE30448000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libc.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;369DE0E1D833CAA693AF17F17C83BA937F0A4DAD&quot; }, { &quot;b&quot; : &quot;7FEE3022D000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libresolv.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;81EF82040E9877E63ADCA93B365F52A4BB831EE1&quot; }, { &quot;b&quot; : &quot;7FEE2FFEC000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libgssapi.so.3&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;40315538B8F467FA9379C26806C2C2E5F75DF62A&quot; }, { &quot;b&quot; : &quot;7FEE2FCBB000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libgnutls.so.30&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;424060547E784D4275171E91AA31276D4A43D1FF&quot; }, { &quot;b&quot; : &quot;7FEE2F9E9000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libkrb5.so.3&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;D60D5A759577E5C477314520EF3D13BDBAED5111&quot; }, { &quot;b&quot; : &quot;7FEE2F7BA000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libk5crypto.so.3&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;09FA442048F4AA6E701F0B504FEB688F09618E30&quot; }, { &quot;b&quot; : &quot;7FEE2F5B6000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libcom_err.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;1E16CB57F699E215A2A8D4EFEF90883BC749B12D&quot; }, { &quot;b&quot; : &quot;7FEE2F3AB000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libkrb5support.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;00DFFD025B1AEBFD619FAE9DE01610BA6F209367&quot; }, { &quot;b&quot; : &quot;7FEE32664000&quot;, &quot;path&quot; : &quot;/lib64/ld-linux-x86-64.so.2&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;EDFA6D46E00CA97F349FDD3333D88493D442932C&quot; }, { &quot;b&quot; : &quot;7FEE2F1A2000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libheimntlm.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;D72147C9E81DC821D2C5CA3A93757EE00DAC67EA&quot; }, { &quot;b&quot; : &quot;7FEE2EF18000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libkrb5.so.26&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;889221E5B7DFAA3D74CD8D55DAEF89DD962DAEB7&quot; }, { &quot;b&quot; : &quot;7FEE2EC76000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libasn1.so.8&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;B7A0E46FB5107A9566AB0D448C865DB7BB1390BF&quot; }, { &quot;b&quot; : &quot;7FEE2EA43000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libhcrypto.so.4&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;54027CA597C9AFC9D32A3D677857E5C8512C9FBF&quot; }, { &quot;b&quot; : &quot;7FEE2E82D000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libroken.so.18&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;AE98B3C2FEC7AEDB3064E03BD8C1883FD12898D1&quot; }, { &quot;b&quot; : &quot;7FEE2E613000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libz.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;340B7B463F981B8A0FB3451751F881DF1B0C2F74&quot; }, { &quot;b&quot; : &quot;7FEE2E3AE000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libp11-kit.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;2F170DCF36786FE91EB82D547A42B2DEE2E9DF81&quot; }, { &quot;b&quot; : &quot;7FEE2E17B000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libidn.so.11&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;F3945D210B52CC973C619CCD08C4394BBC33E07F&quot; }, { &quot;b&quot; : &quot;7FEE2DF68000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libtasn1.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;F5B37E248C55FD582ECD9B0240F96E8458F8E6F0&quot; }, { &quot;b&quot; : &quot;7FEE2DD32000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libnettle.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;74A0B167DA794B5E8B6DAA64367F6385DECF9A6C&quot; }, { &quot;b&quot; : &quot;7FEE2DAFF000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libhogweed.so.4&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;25494D1AE45F2D8F1770895BB899771547668E20&quot; }, { &quot;b&quot; : &quot;7FEE2D87F000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libgmp.so.10&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;7B3533D5998D20EE1A1BE3F87789B69041E7F620&quot; }, { &quot;b&quot; : &quot;7FEE2D67B000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libkeyutils.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;3364D4BF2113C4E8D17EF533867ECC99A53413D6&quot; }, { &quot;b&quot; : &quot;7FEE2D452000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libwind.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;FEA2A08A986C1C954F7DED5A27616F0B4B18C6A2&quot; }, { &quot;b&quot; : &quot;7FEE2D243000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libheimbase.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;E5BC16E04FC04459D5AB8F045B5F7C7838B86027&quot; }, { &quot;b&quot; : &quot;7FEE2CFF8000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libhx509.so.5&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;6B6EF07C8D0FA1CE7CCEE58043BDB6C347F25544&quot; }, { &quot;b&quot; : &quot;7FEE2CD23000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libsqlite3.so.0&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;D9782BA023CAEC26B15D8676E3A5D07B55E121EF&quot; }, { &quot;b&quot; : &quot;7FEE2CAEB000&quot;, &quot;path&quot; : &quot;/lib/x86_64-linux-gnu/libcrypt.so.1&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;1B46E1518CBF3AB34A3FE34E0AFE928AD689BD78&quot; }, { &quot;b&quot; : &quot;7FEE2C8E3000&quot;, &quot;path&quot; : &quot;/usr/lib/x86_64-linux-gnu/libffi.so.6&quot;, &quot;elfType&quot; : 3, &quot;buildId&quot; : &quot;9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F&quot; } ] }}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765|  mongos(__interceptor_backtrace+0x41) [0x55e8678048d1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765|  mongos(_ZN5mongo15printStackTraceERSo+0x113) [0x55e86905b0b3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765|  mongos(+0x2145CA3) [0x55e869059ca3]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765|  libpthread.so.0(+0x113D0) [0x7fee30a383d0]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.654+0000 s20765|  libc.so.6(gsignal+0x38) [0x7fee3047d418]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.655+0000 s20765|  libc.so.6(abort+0x16A) [0x7fee3047f01a]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.655+0000 s20765|  mongos(_ZN5mongo15invariantFailedEPKcS1_j+0xB61) [0x55e868e68bf1]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.655+0000 s20765|  mongos(_ZN5mongo8executor22ThreadPoolTaskExecutor18signalEvent_inlockERKNS0_12TaskExecutor11EventHandleESt11unique_lockISt5mutexE+0x541) [0x55e8687cd021]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.655+0000 s20765|  mongos(_ZN5mongo8executor22ThreadPoolTaskExecutor11signalEventERKNS0_12TaskExecutor11EventHandleE+0xFE) [0x55e8687d088e]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  mongos(_ZN5mongo18AsyncResultsMerger4killEv+0xC6B) [0x55e868d0e07b]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  mongos(_ZN5mongo16RouterStageMerge4killEv+0x94) [0x55e868d3f724]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  mongos(_ZN5mongo20ClusterCursorManager17reapZombieCursorsEv+0xC0D) [0x55e868d1d7ed]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  mongos(_ZN5mongo23ClusterCursorCleanupJob3runEv+0x1A4) [0x55e868d19a54]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  mongos(_ZN5mongo13BackgroundJob7jobBodyEv+0x3BE) [0x55e868e789ce]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  libstdc++.so.6(+0xB8C80) [0x7fee30cfcc80]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.656+0000 s20765|  libpthread.so.0(+0x76FA) [0x7fee30a2e6fa]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.657+0000 s20765|  libc.so.6(clone+0x6D) [0x7fee3054eb5d]&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:views_all_commands] 2016-08-28T18:23:49.657+0000 s20765| -----  END BACKTRACE  -----&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This crash occurs due to the following sequence of events:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The &lt;tt&gt;ClusterCursorCleanupJob&lt;/tt&gt; kills a cursor that is pending for deletion. During cursor killing, it &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc1/src/mongo/s/query/async_results_merger.cpp#L629&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;asks the ThreadPoolTaskExecutor for an event&lt;/a&gt; which will be signaled when it is safe to destroy the cursor.&lt;/li&gt;
	&lt;li&gt;This happens at a time when the mongos process, and hence the &lt;tt&gt;ThreadPoolTaskExecutor&lt;/tt&gt;, is not shutting down. This means we skip &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc1/src/mongo/s/query/async_results_merger.cpp#L629&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this if-block&lt;/a&gt; and hold onto the event as &lt;tt&gt;AsyncResultsMerger::_killCursorsScheduledEvent&lt;/tt&gt;.&lt;/li&gt;
	&lt;li&gt;Before we get a chance to signal the event, but after it has been successfully created, the mongos process begins its shutdown sequence. As part of this process, we call &lt;tt&gt;ThreadPoolTaskExecutor::join()&lt;/tt&gt;, which &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc1/src/mongo/executor/thread_pool_task_executor.cpp#L163-L170&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;signals any unsignaled events&lt;/a&gt;.&lt;/li&gt;
	&lt;li&gt;The killing thread is unaware of this, and happily proceeds to &lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc1/src/mongo/s/query/async_results_merger.cpp#L646&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;signal the event again&lt;/a&gt;. Oops!&lt;/li&gt;
&lt;/ol&gt;
</comment>
                            <comment id="1403462" author="david.storch" created="Fri, 7 Oct 2016 22:10:20 +0000"  >&lt;p&gt;The cause behind the invariant failure described by &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=schwerin&quot; class=&quot;user-hover&quot; rel=&quot;schwerin&quot;&gt;schwerin&lt;/a&gt; is not yet clear to me, though I can see from code inspection how the originally reported invariant failure can occur. The original invariant failure happens in RouterStageMerge::kill(), which asks the AsyncResultsMerger for an event and then waits on it:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc0/src/mongo/s/query/router_stage_merge.cpp#L59-L60&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.4.0-rc0/src/mongo/s/query/router_stage_merge.cpp#L59-L60&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If mongos shutdown is in progress, however, we will have returned an invalid Event:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.4.0-rc0/src/mongo/s/query/async_results_merger.cpp#L635&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/r3.4.0-rc0/src/mongo/s/query/async_results_merger.cpp#L635&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We trip the invariant when attempting to wait on this invalid Event. The bug should only manifest when a mongos cursor gets killed (or kills itself) concurrently with mongos shutdown. The stack trace provided by Andy seems to be a separate bug, probably with a very similar root cause.&lt;/p&gt;</comment>
                            <comment id="1403458" author="schwerin" created="Fri, 7 Oct 2016 22:05:09 +0000"  >&lt;p&gt;The stack trace from the repro instructions is different from the one in the original bug report. In the repro, the invalid event is returned from AsyncResultsMerger::nextEvent, and in the original report it comes from AsyncResultsMerger::kill. Maybe two similar bugs, maybe one root cause. Here&apos;s the stack trace from the repro, as of 3.4.0-rc0:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  * frame #0: 0x0000555556c1f812 mongos`mongo::executor::ThreadPoolTaskExecutor::waitForEvent(this=0x00007ffff7f1d980, event=0x00007ffff7eb44a0) + 32 at thread_pool_task_executor.cpp:234&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #1: 0x0000555556da47c7 mongos`mongo::RouterStageMerge::next(this=0x00007ffff1634220) + 237 at router_stage_merge.cpp:52&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #2: 0x0000555556d8fc0b mongos`mongo::ClusterClientCursorImpl::next(this=0x00007ffff185b6a0) + 281 at cluster_client_cursor_impl.cpp:84&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #3: 0x0000555556d9f8a0 mongos`mongo::(anonymous namespace)::runQueryWithoutRetrying(txn=0x00007fffdd73d4a0, query=0x00007ffff1601f80, readPref=0x00007ffff7eb4eb0, chunkManager=0x0000000000000000, primary=shared_ptr&amp;lt;mongo::Shard&amp;gt; @ 0x00007ffff7eb4ab0, results=0x00007ffff7eb4ce0, viewDefinition=0x00007ffff7eb4c90) + 2597 at cluster_find.cpp:236&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #4: 0x0000555556da0466 mongos`mongo::ClusterFind::runQuery(txn=0x00007fffdd73d4a0, query=0x00007ffff1601f80, readPref=0x00007ffff7eb4eb0, results=0x00007ffff7eb4ce0, viewDefinition=0x00007ffff7eb4c90) + 946 at cluster_find.cpp:333&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #5: 0x0000555556d23fa4 mongos`mongo::(anonymous namespace)::ClusterFindCmd::run(this=0x0000555558170860, txn=0x00007fffdd73d4a0, dbname=&quot;PP???\x7f&quot;..., cmdObj=0x00007ffff7eb53d0, options=0, errmsg=&quot;\xb0P???\x7f&quot;, result=0x00007ffff7eb54c0) const + 992 at cluster_find_cmd.cpp:195&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #6: 0x0000555556db0ffe mongos`mongo::Command::execCommandClient(txn=0x00007fffdd73d4a0, c=0x0000555558170860, queryOptions=0, ns=&quot;test.$cmd&quot;, cmdObj=0x00007ffff7eb53d0, result=0x00007ffff7eb54c0) + 1190 at s_only.cpp:141&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #7: 0x0000555556d6c9a5 mongos`mongo::(anonymous namespace)::runAgainstRegistered(txn=0x00007fffdd73d4a0, ns=&quot;test.$cmd&quot;, jsobj=0x00007ffff7eb53d0, anObjBuilder=0x00007ffff7eb54c0, queryOptions=0) + 660 at strategy.cpp:111&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #8: 0x0000555556d6e5af mongos`mongo::Strategy::clientCommandOp(txn=0x00007fffdd73d4a0, request=0x00007ffff7eb5910) + 1995 at strategy.cpp:324&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #9: 0x0000555556d69579 mongos`mongo::Request::process(this=0x00007ffff7eb5910, txn=0x00007fffdd73d4a0, attempt=0) + 849 at request.cpp:110&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #10: 0x0000555556615c72 mongos`mongo::ServiceEntryPointMongos::_sessionLoop(this=0x00007ffff118c8a0, session=0x00007fffef95baa0) + 488 at service_entry_point_mongos.cpp:114&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #11: 0x00005555566159f0 mongos`operator(__closure=0x00007fffef95bb28, session=0x00007fffef95baa0) + 38 at service_entry_point_mongos.cpp:78&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #12: 0x0000555556616521 mongos`std::_Function_handler&amp;lt;void(mongo::transport::Session*), mongo::ServiceEntryPointMongos::startSession(mongo::transport::Session&amp;amp;&amp;amp;)::&amp;lt;lambda(mongo::transport::Session*)&amp;gt; &amp;gt;::_M_invoke(__functor=0x00007fffef95bb28, __args#0=0x00007ffff7eb5b60) + 55 at functional:1871&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #13: 0x0000555556e0fc79 mongos`std::function&amp;lt;void (mongo::transport::Session*)&amp;gt;::operator(this=0x00007fffef95bb28, __args#0=0x00007fffef95baa0)(mongo::transport::Session*) const + 73 at functional:2267&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #14: 0x0000555556e0ebed mongos`mongo::(anonymous namespace)::runFunc(ptr=0x00007fffef95baa0) + 399 at service_entry_point_utils.cpp:74&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #15: 0x00007ffff5fa26fa libpthread.so.0`start_thread + 202&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;    frame #16: 0x00007ffff5cd8b5d libc.so.6`__clone + 109 at clone.S:109&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                            <comment id="1403448" author="schwerin" created="Fri, 7 Oct 2016 21:54:03 +0000"  >&lt;p&gt;This appears to be a race between the AsyncResultsMerger&apos;s execution of nextEvent() and shutting down the task executor, causing nextEvent() to return Status::OK() with an invalid event. This causes the RouterStageMerge processor to call wait on an invalid event, violating the invariant.&lt;/p&gt;

&lt;p&gt;Assigning to query team for analysis and triage. I&apos;m pretty sure this only happens during mongos shutdown.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="503086">SERVER-33522</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="339880">SERVER-27460</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="44604">SERVER-6496</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="24879">SERVER-4274</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>5.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[500A000000X37A2IAJ, 500A000000aAmtRIAS]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 9 Aug 2016 12:06:02 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 16 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>nuno.costa@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 16 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>15.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>schwerin@mongodb.com</customfieldvalue>
            <customfieldvalue>davenson.lombard@mongodb.com</customfieldvalue>
            <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>ian.boros@mongodb.com</customfieldvalue>
            <customfieldvalue>ruoxin.xu@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjzr3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hua4pb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1235">Sharding 2016-10-10</customfieldvalue>
    <customfieldvalue id="2120">Query 2018-02-26</customfieldvalue>
    <customfieldvalue id="3499">Query 2020-05-04</customfieldvalue>
    <customfieldvalue id="3616">Query 2020-10-05</customfieldvalue>
    <customfieldvalue id="3617">Query 2020-10-19</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hseovz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>