<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:39:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-35425] After a map reduce an exception NamespaceNotFound happens in a secondary shard</title>
                <link>https://jira.mongodb.org/browse/SERVER-35425</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Can you help with the following problem.&lt;/p&gt;

&lt;p&gt;When I do a map reduce, after it is finished... i get the following expection:&lt;/p&gt;

&lt;p&gt;2018-06-05T14:52:53.464+0000 F REPL&#160;&#160;&#160;&#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 3&amp;#93;&lt;/span&gt; writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts&lt;/p&gt;

&lt;p&gt;2018-06-05T14:52:53.464+0000 F -&#160;&#160;&#160;&#160;&#160;&#160;&#160; &lt;span class=&quot;error&quot;&gt;&amp;#91;repl writer worker 3&amp;#93;&lt;/span&gt; Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): {&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I saw that you have some Jira issues regarding this exception NamespaceNotFound, but all are&#160; already closed, i could not figure out what was the solution for this.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Thank you&lt;/p&gt;</description>
                <environment></environment>
        <key id="554746">SERVER-35425</key>
            <summary>After a map reduce an exception NamespaceNotFound happens in a secondary shard</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="esha.maharishi@mongodb.com">Esha Maharishi</assignee>
                                    <reporter username="ruiribeiro">Rui Ribeiro</reporter>
                        <labels>
                    </labels>
                <created>Tue, 5 Jun 2018 22:14:28 +0000</created>
                <updated>Thu, 12 Jul 2018 17:18:20 +0000</updated>
                            <resolved>Thu, 12 Jul 2018 17:18:00 +0000</resolved>
                                    <version>3.6.4</version>
                                                    <component>Replication</component>
                                        <votes>0</votes>
                                    <watches>11</watches>
                                                                                                                <comments>
                            <comment id="1945784" author="esha.maharishi@10gen.com" created="Thu, 12 Jul 2018 17:17:43 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ruiribeiro&quot; class=&quot;user-hover&quot; rel=&quot;ruiribeiro&quot;&gt;ruiribeiro&lt;/a&gt;, I am going to close this as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt;, but let me know if you are able to provide the additional info and want a more specific diagnosis.&lt;/p&gt;</comment>
                            <comment id="1942620" author="esha.maharishi@10gen.com" created="Tue, 10 Jul 2018 00:36:11 +0000"  >&lt;p&gt;I think most likely, inserts from a migration on &lt;tt&gt;M_20180602.tmp.uimsi_d_a3&lt;/tt&gt; (which is &lt;b&gt;not&lt;/b&gt; one of the internal collections created by mapReduce) are colliding with dropping the tmp.mr namespaces of mapReduces on the destination shard.&lt;/p&gt;

&lt;p&gt;In this case, you should be able to avoid this specific manifestation of the bug by turning off the balancer.&lt;/p&gt;</comment>
                            <comment id="1942615" author="esha.maharishi@10gen.com" created="Tue, 10 Jul 2018 00:01:18 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ruiribeiro&quot; class=&quot;user-hover&quot; rel=&quot;ruiribeiro&quot;&gt;ruiribeiro&lt;/a&gt;, you mentioned on&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt; you are using a multi-mapReduce process:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;I understand your solution, but i am not using the option out: &quot;replace&quot;, I use the option out: &quot;reduce&quot;&lt;/p&gt;

&lt;p&gt;In my situation I do:&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;Pickup 3 Collections (sharded)&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#166;&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; Map Reduce (out:reduce, sharded:true)&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#166;&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; Ouput Collection(MR 1)&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#166;&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160;Map Reduce (out:reduce, sharded:true)&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#166;&lt;/p&gt;

&lt;p&gt;&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; Ouput Collection(MR 2)&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;1) Can you list the &lt;ins&gt;exact mapReduce commands&lt;/ins&gt; (specifically, including the input and output namespaces) used in this multi-step process?&lt;/p&gt;

&lt;p&gt;I ask because it seems like some of your &lt;b&gt;user-created collections&lt;/b&gt;&#160;(including those created by specifying them as the $out collection of a mapReduce) start with &quot;tmp&quot;. This is confusing at first, because mapReduce also creates internal collections that start with &quot;tmp&quot; (&quot;tmp.mr&quot; and &quot;tmp.mrs&quot;).&lt;/p&gt;

&lt;p&gt;2) Are you also running migrations on your &lt;b&gt;user-created collection that starts with &quot;tmp&quot;&lt;/b&gt; (or, do you have the balancer on?)&lt;/p&gt;

&lt;p&gt;I think you are hitting some variant of the bug in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt;, but am not sure exactly how it&apos;s manifesting.&lt;/p&gt;</comment>
                            <comment id="1924279" author="spencer" created="Mon, 18 Jun 2018 22:04:05 +0000"  >&lt;p&gt;Yeah, the node is definitely SECONDARY when the crash happens.&lt;/p&gt;

&lt;p&gt;Something weird from the logs is that a few seconds before the crash happens, a collection with the same UUID but a different namespace is dropped:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:45:03.169+0000 I COMMAND  [repl writer worker 4] CMD: drop M_20180602.tmp.mr.uimsi_d_6&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:45:03.169+0000 I STORAGE  [repl writer worker 4] dropCollection: M_20180602.tmp.mr.uimsi_d_6 (16e14cce-7a36-4e3d-850c-0e630dedaea3) - renaming to drop-pending collection: M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 with drop optime { ts: Timestamp(1528209903, 23797), t: 6 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:45:03.245+0000 I REPL     [replication-423] Completing collection drop for M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 with drop optime { ts: Timestamp(1528209903, 23797), t: 6 } (notification optime: { ts: Timestamp(1528209903, 23797), t: 6 })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:45:03.245+0000 I STORAGE  [replication-423] Finishing collection drop for M_20180602.system.drop.1528209903i23797t6.tmp.mr.uimsi_d_6 (16e14cce-7a36-4e3d-850c-0e630dedaea3).&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:52:53.464+0000 F REPL     [repl writer worker 3] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a&quot;, vpmn: &quot;TZAVC&quot;, hpmn: &quot;KENSA&quot; }, value: { msisdn: &quot;2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1&quot;, imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } } on: { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a&quot;, vpmn: &quot;TZAVC&quot;, hpmn: &quot;KENSA&quot; }, value: { msisdn: &quot;2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1&quot;, imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:52:53.464+0000 F -        [repl writer worker 3] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 667), t: 6, h: 285159385916138329, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e783793e1a0491a44cd352be6608835596895d1da3e5197d2cf288f6e4b85a&quot;, vpmn: &quot;TZAVC&quot;, hpmn: &quot;KENSA&quot; }, value: { msisdn: &quot;2e045846194e8240ff95e78d3d8ad9358182f8c068cfd1b0d73711e18e4e19b1&quot;, imei: undefined, hsai_s: 14.0, hul_s: 13.0, hcl_s: 14.0, hcl_f: 13.0, hmtfw_s: 1.0, hmtfw_t: 2.0, hprn_s: 4.0 } } } at src/mongo/db/repl/sync_tail.cpp 1204&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:52:53.464+0000 F -        [repl writer worker 3] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This does sound a lot like the description of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt;.  Sending back to sharding to rethink whether this could be another manifestation of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1924186" author="esha.maharishi@10gen.com" created="Mon, 18 Jun 2018 21:05:05 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt;, can you confirm from the logs attached to the ticket that this node is in steady state replication when the crash happens, not initial sync?&lt;/p&gt;</comment>
                            <comment id="1924165" author="esha.maharishi@10gen.com" created="Mon, 18 Jun 2018 20:59:29 +0000"  >&lt;p&gt;Actually, that&apos;s weird, how could there even be a migration active on a temp collection?! A migration is tied to a particular collection...&lt;/p&gt;</comment>
                            <comment id="1924164" author="esha.maharishi@10gen.com" created="Mon, 18 Jun 2018 20:58:39 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=spencer&quot; class=&quot;user-hover&quot; rel=&quot;spencer&quot;&gt;spencer&lt;/a&gt; that makes sense since these oplog entries were marked &apos;fromMigrate: true&apos;.&lt;/p&gt;

&lt;p&gt;I think we should simply not include writes to temp collections in migrations.&lt;/p&gt;

&lt;p&gt;I am curious though why this only seems to trip an assert in 3.6, since this seems like a beginning-of-time bug.&lt;/p&gt;</comment>
                            <comment id="1924161" author="spencer" created="Mon, 18 Jun 2018 20:56:00 +0000"  >&lt;p&gt;Swallowing the NamespaceNotFound error during oplog application only happens for initial sync and startup recovery. In steady state, we should not need to ignore NamespaceNotFound errors as they are not expected.  This means that somehow there was an &apos;insert&apos; oplog entry for this namespace without there being a &apos;create&apos; oplog entry for it.&lt;/p&gt;</comment>
                            <comment id="1918307" author="esha.maharishi@10gen.com" created="Tue, 12 Jun 2018 14:12:49 +0000"  >&lt;p&gt;I am going to move this to the repl backlog as Needs Triage.&lt;/p&gt;</comment>
                            <comment id="1918080" author="rribeiro" created="Tue, 12 Jun 2018 08:56:16 +0000"  >&lt;p&gt;Thank you&#160;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt;&#160; for the help.&lt;/p&gt;

&lt;p&gt;Yes the problem it is happening&#160; in a secondary node and that collection is the temporary collection that the Map Reduce creates.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="1917268" author="esha.maharishi@10gen.com" created="Mon, 11 Jun 2018 17:43:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=rribeiro&quot; class=&quot;user-hover&quot; rel=&quot;rribeiro&quot;&gt;rribeiro&lt;/a&gt;, I took a quick look at the logs attached here based on your comment on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;It looks like the node is a secondary, and the crash occurs during regular replset syncing (since the crash occurs in sync_tail.cpp).&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.508+0000 I REPL     [replexec-0] This node is cmodb889.abc.com:27018 in the config&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.508+0000 I REPL     [replexec-0] transition to STARTUP2 from STARTUP&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.509+0000 I REPL     [replexec-0] transition to RECOVERING from STARTUP2&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.510+0000 I REPL     [rsSync] transition to SECONDARY from RECOVERING&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.537+0000 I REPL     [replexec-2] Member cmodb881.abc.com:27018 is now in state PRIMARY&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.541+0000 I REPL     [replexec-1] Member cmodb875.abc.com:27027 is now in state ARBITER&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:39.583+0000 I REPL     [replexec-0] Member cmodb899.abc.com:27027 is now in state ARBITER&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.510+0000 I REPL     [rsBackgroundSync] sync source candidate: cmodb881.abc.com:27018&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.510+0000 I ASIO     [NetworkInterfaceASIO-RS-0] Connecting to cmodb881.abc.com:27018&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.529+0000 I ASIO     [NetworkInterfaceASIO-RS-0] Successfully connected to cmodb881.abc.com:27018, took 19ms (1 connections now open to cmodb881.abc.com:27018)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;...&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// NamespaceNotFound exception triggered by oplog entry with &apos;fromMigrate: true&apos; for an insert (&apos;op: &quot;i&quot;&apos;) into M_20180602.tmp.uimsi_d_a3&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.871+0000 F REPL     [repl writer worker 10] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f&quot;, vpmn: &quot;SGPM1&quot;, hpmn: &quot;CHNCT&quot; }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } } on: { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f&quot;, vpmn: &quot;SGPM1&quot;, hpmn: &quot;CHNCT&quot; }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// NamespaceNotFound trips an fassert&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.871+0000 F -        [repl writer worker 10] Fatal assertion 16359 NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 684), t: 6, h: -5869453682661485531, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7c101c004f89624c28e364b783ee821361149287cc59d271c982feac1cc8f&quot;, vpmn: &quot;SGPM1&quot;, hpmn: &quot;CHNCT&quot; }, value: { msisdn: undefined, imei: undefined, vsai_f: 2.0, vgsai_f: 2.0 } } } at src/mongo/db/repl/sync_tail.cpp 1204&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.871+0000 F REPL     [repl writer worker 15] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0&quot;, vpmn: &quot;GBRVF&quot;, hpmn: &quot;SWEIQ&quot; }, value: { msisdn: &quot;da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07&quot;, imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } } on: { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0&quot;, vpmn: &quot;GBRVF&quot;, hpmn: &quot;SWEIQ&quot; }, value: { msisdn: &quot;da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07&quot;, imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;// same occurs on another &apos;repl writer worker&apos; thread, for an oplog entry for an insert on the same collection&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.871+0000 F REPL     [repl writer worker 15] writer worker caught exception: NamespaceNotFound: Failed to apply operation due to missing collection (16e14cce-7a36-4e3d-850c-0e630dedaea3): { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0&quot;, vpmn: &quot;GBRVF&quot;, hpmn: &quot;SWEIQ&quot; }, value: { msisdn: &quot;da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07&quot;, imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } } on: { ts: Timestamp(1528210373, 697), t: 6, h: 140069530794967834, v: 2, op: &quot;i&quot;, ns: &quot;M_20180602.tmp.uimsi_d_a3&quot;, ui: UUID(&quot;16e14cce-7a36-4e3d-850c-0e630dedaea3&quot;), fromMigrate: true, wall: new Date(1528210373457), o: { _id: { utc: new Date(1527897600000), imsi: &quot;46e7e2ad3736701706d1c233286baa456e2128dcb15066f622938dfb28060ac0&quot;, vpmn: &quot;GBRVF&quot;, hpmn: &quot;SWEIQ&quot; }, value: { msisdn: &quot;da2c302c05b2417c9aeb2b8cf3e196700ef6d9120cae1aa5538fb4fb59d8cb07&quot;, imei: undefined, hsai_s: 5.0, hgsai_s: 4.0, hul_s: 1.0, hgul_s: 1.0 } } }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;2018-06-05T14:53:40.871+0000 F -        [repl writer worker 15] &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&amp;nbsp;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;***aborting after fassert() failure &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;It seems like two of the &quot;repl writer worker&quot; threads (10 and 15) encountered a NamespaceNotFound exception (&quot;writer worker caught exception: NamesapceNotFound&quot;) on an insert oplog entry for the&#160;&lt;tt&gt;M_20180602.tmp.uimsi_d_a3&lt;/tt&gt; collection.&lt;/p&gt;

&lt;p&gt;(This collection should be one of the tmp collections used for writing the intermediate results of a mapReduce; I also notice that both oplog entries that triggered the exception had &lt;tt&gt;from migrate: true).&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;It looks like&#160;&lt;a href=&quot;https://github.com/mongodb/mongo/blob/r3.6.5/src/mongo/db/repl/sync_tail.cpp#L1374-L1378&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this was the fassert tripped (on v3.6.5)&lt;/a&gt;&#160;in &lt;tt&gt;multiSyncApply()&lt;/tt&gt; , and that in &lt;a href=&quot;https://github.com/mongodb/mongo/commit/5cbbdd3e016d6a53ca3ce035762017f005c05d7f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this commit on v4.0&lt;/a&gt;&#160;by for&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-32334&quot; title=&quot;Update startup/rollback recovery to use new Applier interface for oplog application&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-32334&quot;&gt;&lt;del&gt;SERVER-32334&lt;/del&gt;&lt;/a&gt;, special handling was added to ignore the NamespaceNotFound error code. (Also see&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24927&quot; title=&quot;Ignore NamespaceNotFound errors during initial sync apply ops, because they are transient&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24927&quot;&gt;&lt;del&gt;SERVER-24927&lt;/del&gt;&lt;/a&gt;, mentioned in the comment in the commit).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=benety.goh&quot; class=&quot;user-hover&quot; rel=&quot;benety.goh&quot;&gt;benety.goh&lt;/a&gt;, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=judah.schvimer&quot; class=&quot;user-hover&quot; rel=&quot;judah.schvimer&quot;&gt;judah.schvimer&lt;/a&gt;, should ignoring NamespaceNotFound in this catch block be backported to 3.6?&lt;/p&gt;</comment>
                            <comment id="1915265" author="ruiribeiro" created="Fri, 8 Jun 2018 15:36:07 +0000"  >&lt;p&gt;Hi&lt;/p&gt;

&lt;p&gt;I just updated the mongo log of the shard where this fatal assertion happens.&lt;/p&gt;

&lt;p&gt;This problem impacted a client apliccation that&#160;was blocked for inserts after the failure.&lt;/p&gt;

&lt;p&gt;Even I have a replica set of two members and one arbiter.&lt;/p&gt;</comment>
                            <comment id="1912270" author="kaloian.manassiev" created="Wed, 6 Jun 2018 13:10:23 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ruiribeiro&quot; class=&quot;user-hover&quot; rel=&quot;ruiribeiro&quot;&gt;ruiribeiro&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for your report.&lt;/p&gt;

&lt;p&gt;Would it be possible to attach the complete mongod logs from the shard which experienced this fatal assertion?&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
-Kal.&lt;/p&gt;</comment>
                            <comment id="1912268" author="kaloian.manassiev" created="Wed, 6 Jun 2018 13:09:03 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=janna.golden&quot; class=&quot;user-hover&quot; rel=&quot;janna.golden&quot;&gt;janna.golden&lt;/a&gt;/&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=esha.maharishi&quot; class=&quot;user-hover&quot; rel=&quot;esha.maharishi&quot;&gt;esha.maharishi&lt;/a&gt; - is it possible that this is a manifestation of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-33538&quot; title=&quot;mapReduce &amp;quot;replace&amp;quot; on a sharded output collection can lead to UUIDCatalog inconsistencies&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-33538&quot;&gt;&lt;del&gt;SERVER-33538&lt;/del&gt;&lt;/a&gt; or something related?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="503499">SERVER-33538</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="188992" name="mongod.log" size="78930" author="ruiribeiro" created="Fri, 8 Jun 2018 15:32:45 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>3.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 6 Jun 2018 13:09:03 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 30 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 30 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>esha.maharishi@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>rribeiro</customfieldvalue>
            <customfieldvalue>ruiribeiro</customfieldvalue>
            <customfieldvalue>spencer@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htzuxj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr8l7j:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2297">Repl 2018-06-18</customfieldvalue>
    <customfieldvalue id="2385">Sharding 2018-07-02</customfieldvalue>
    <customfieldvalue id="2386">Sharding 2018-07-16</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htzh6v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>