<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:47:09 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-59403] upgrade from 4.4.4 to 4.4.8,secondary crash indicates _id duplicate key </title>
                <link>https://jira.mongodb.org/browse/SERVER-59403</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;my cluster is 4.4.8 with three shards.&lt;br/&gt;
cluster upgrade from 4.4.4 to 4.4.8. cluster is ok.after one hours,there is secondary crash ,indicates _id duplicate key .&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T18:54:43.182+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:21238,   &quot;ctx&quot;:&quot;ReplWriterWorker-0&quot;,&quot;msg&quot;:&quot;Writer worker caught exception&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;DuplicateKey{ keyPattern: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, keyValue: &lt;/p&gt;
{ _id: \&quot;0be47ed91b91474f8a96429fa7d3cfec\&quot; }
&lt;p&gt; }: E11000 duplicate key error collection: customerService.TrackerDetail index: &lt;em&gt;id&lt;/em&gt; dup key: &lt;/p&gt;
{ _id: \&quot;0be47ed91b91474f8a96429fa7d3cfec\&quot; }
&lt;p&gt;&quot;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1853019">SERVER-59403</key>
            <summary>upgrade from 4.4.4 to 4.4.8,secondary crash indicates _id duplicate key </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="eric.sedor@mongodb.com">Eric Sedor</assignee>
                                    <reporter username="601290552@qq.com">jing xu</reporter>
                        <labels>
                    </labels>
                <created>Tue, 17 Aug 2021 11:09:17 +0000</created>
                <updated>Wed, 7 Sep 2022 14:46:07 +0000</updated>
                            <resolved>Mon, 18 Oct 2021 17:52:16 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>8</watches>
                                                                                                                <comments>
                            <comment id="4130247" author="eric.sedor" created="Mon, 18 Oct 2021 17:51:49 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=601290552%40qq.com&quot; class=&quot;user-hover&quot; rel=&quot;601290552@qq.com&quot;&gt;601290552@qq.com&lt;/a&gt; I&apos;m going to close this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7995&quot; title=&quot;Fix the global visibility so that it cannot go beyond checkpoint visibility&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7995&quot;&gt;&lt;del&gt;WT-7995&lt;/del&gt;&lt;/a&gt;. Please comment there if you end up needing clarification on the remediation steps we provided.&lt;/p&gt;</comment>
                            <comment id="4101042" author="eric.sedor" created="Mon, 4 Oct 2021 22:03:07 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=601290552%40qq.com&quot; class=&quot;user-hover&quot; rel=&quot;601290552@qq.com&quot;&gt;601290552@qq.com&lt;/a&gt;, I wanted to clarify: it is possible for &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7995&quot; title=&quot;Fix the global visibility so that it cannot go beyond checkpoint visibility&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7995&quot;&gt;&lt;del&gt;WT-7995&lt;/del&gt;&lt;/a&gt; or &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7984&quot; title=&quot;Fix a bug that could cause a checkpoint to omit a page of data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7984&quot;&gt;&lt;del&gt;WT-7984&lt;/del&gt;&lt;/a&gt; to persist inconsistencies that cause a crash later, not just on startup.&lt;/p&gt;

&lt;p&gt;So we recommend upgrading to 4.4.9 and performing the remediation steps suggested in those tickets. Does that make sense?&lt;/p&gt;</comment>
                            <comment id="4084640" author="601290552@qq.com" created="Sat, 25 Sep 2021 12:50:48 +0000"  >&lt;p&gt;Hi eric:&lt;br/&gt;
   i read this two issue about duplicate key.it indicates: db crash or unclean shutdown.&lt;br/&gt;
but i shutdown db using normal not abort.&lt;/p&gt;

&lt;p&gt;secondary:&lt;br/&gt;
The bug can cause a Duplicate Key error on startup and prevent the node from starting.&lt;/p&gt;

&lt;p&gt;my 4.4.8 secondary can startup normal,it is runing a few hours that indicates &lt;br/&gt;
Duplicate Key error not startup.&lt;/p&gt;</comment>
                            <comment id="4081047" author="eric.sedor" created="Thu, 23 Sep 2021 16:18:58 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=601290552%40qq.com&quot; class=&quot;user-hover&quot; rel=&quot;601290552@qq.com&quot;&gt;601290552@qq.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;m very sorry for the delay. I believe you should upgrade to 4.4.9 (just released) as it is very likely from the occurrence of duplicate key errors that you&apos;ve been impacted by wither &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7995&quot; title=&quot;Fix the global visibility so that it cannot go beyond checkpoint visibility&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7995&quot;&gt;&lt;del&gt;WT-7995&lt;/del&gt;&lt;/a&gt; or &lt;a href=&quot;https://jira.mongodb.org/browse/WT-7984&quot; title=&quot;Fix a bug that could cause a checkpoint to omit a page of data&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-7984&quot;&gt;&lt;del&gt;WT-7984&lt;/del&gt;&lt;/a&gt;, which are fixed in this version.&lt;/p&gt;

&lt;p&gt;Eric&lt;/p&gt;</comment>
                            <comment id="4012024" author="601290552@qq.com" created="Fri, 20 Aug 2021 08:45:14 +0000"  >&lt;p&gt;Hi Eric:&lt;br/&gt;
    Have you made any progress for it?thanks.&lt;/p&gt;
</comment>
                            <comment id="4005720" author="601290552@qq.com" created="Wed, 18 Aug 2021 03:36:47 +0000"  >&lt;p&gt;Hi Eric:&lt;br/&gt;
  My cluster version upgrade from 4.4.4 to 4.4.8.cluster has three shards.&lt;br/&gt;
every shard has three members(PSS).i upgrade frist:config-&lt;del&gt;shard&lt;/del&gt;-mongos.i will upload shard2 data.&lt;br/&gt;
upgrade shard2 step:&lt;br/&gt;
   first upgrade all secondary member.i shutdown instance normal.after i use 4.4.8 binary mongo start it.&lt;br/&gt;
when i upgrade primary.i execute rs.stepDown(600).because it has high priority.after primary becomes seconday,i shutdown it normal.&lt;br/&gt;
after  i use 4.4.8 binary mongo start it.After the upgrade is complete, the system runs for more than 8 hours.&lt;/p&gt;

&lt;p&gt;Then, can you elaborate on the timeline of what has occurred here?&lt;br/&gt;
yes,it is ok.&lt;/p&gt;

&lt;p&gt;shard2 seconday is affected nodes(srvdb303.yto.cloud)&lt;br/&gt;
1&#12289;shutdown using 4.4.4 time:2021-08-17T16:12:36.840+08:00&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T16:12:36.840+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23138,   &quot;ctx&quot;:&quot;conn362334&quot;,&quot;msg&quot;:&quot;Shutting down&quot;,&quot;attr&quot;:{&quot;exitCode&quot;:0}}&lt;br/&gt;
2&#12289;startup using 4.4.8 time:2021-08-17T16:12:44.132+08:00&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T16:12:44.132+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:20698,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;***** SERVER RESTARTED *****&quot;}&lt;br/&gt;
3&#12289;crash using 4.4.8 time:2021-08-18T10:07:19.822+08:00&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;from the following log:this duplicate _id is op:u,update lead to this duplicate&amp;#93;&lt;/span&gt;&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T10:07:19.821+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:21238,   &quot;ctx&quot;:&quot;ReplWriterWorker-1777&quot;,&quot;msg&quot;:&quot;Writer worker caught exception&quot;,&lt;br/&gt;
&quot;attr&quot;:{&quot;error&quot;:&quot;DuplicateKey{ keyPattern: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, keyValue: &lt;/p&gt;
{ _id: \&quot;a148c2bd1c274559b674ef3eddb46d01\&quot; }
&lt;p&gt; }: &lt;br/&gt;
E11000 duplicate key error collection: customerService.TrackerDetail index: &lt;em&gt;id&lt;/em&gt; dup key: &lt;/p&gt;
{ _id: \&quot;a148c2bd1c274559b674ef3eddb46d01\&quot; }
&lt;p&gt;&quot;,&lt;br/&gt;
&quot;oplogEntry&quot;:{&quot;op&quot;:&quot;u&quot;,&quot;ns&quot;:&quot;customerService.TrackerDetail&quot;,&quot;ui&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;6345c1d9-8e32-4d82-a2bb-b6c6313f17c1&quot;}
&lt;p&gt;,&lt;br/&gt;
&quot;o&quot;:{&quot;$v&quot;:1,&quot;$set&quot;:{&quot;opTime&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T02:07:19.000Z&quot;}
&lt;p&gt;,&lt;br/&gt;
&quot;opUt&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T02:07:19.824Z&quot;}
&lt;p&gt;}},&quot;o2&quot;:&lt;/p&gt;
{&quot;no&quot;:&quot;xxxx&quot;,&quot;_id&quot;:&quot;a148c2bd1c274559b674ef3eddb46d01&quot;}
&lt;p&gt;,&lt;br/&gt;
&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1629252439,&quot;i&quot;:252}},&quot;t&quot;:8,&quot;v&quot;:2,&quot;wall&quot;:{&quot;$date&quot;:&quot;2021-08-18T02:07:19.819Z&quot;}}}}&lt;/p&gt;


&lt;p&gt;{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T10:07:19.822+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23095,   &quot;ctx&quot;:&quot;OplogApplier-0&quot;,&quot;msg&quot;:&quot;Fatal assertion&quot;,&quot;attr&quot;:{&quot;msgid&quot;:34437,&quot;error&quot;:&quot;DuplicateKey{ keyPattern: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, &lt;br/&gt;
keyValue: &lt;/p&gt;
{ _id: \&quot;a148c2bd1c274559b674ef3eddb46d01\&quot; }
&lt;p&gt; }: E11000 duplicate key error collection: customerService.TrackerDetail index: &lt;em&gt;id&lt;/em&gt; dup key: &lt;/p&gt;
{ _id: \&quot;a148c2bd1c274559b674ef3eddb46d01\&quot; }
&lt;p&gt;&quot;,&quot;file&quot;:&quot;src/mongo/db/repl/oplog_applier_impl.cpp&quot;,&quot;line&quot;:510}}&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T10:07:19.822+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;-&quot;,        &quot;id&quot;:23096,   &quot;ctx&quot;:&quot;OplogApplier-0&quot;,&quot;msg&quot;:&quot;\n\n***aborting after fassert() failure\n\n&quot;}&lt;/p&gt;

&lt;p&gt;1&#12289;The logs for the affected nodes, including before, leading up to, and after the first sign of corruption.&lt;br/&gt;
2&#12289;a clear timeline of activities (with timestamps and timezones) including server restarts, replica set changes, version changes, etc.&lt;br/&gt;
1&#12289;shutdown using 4.4.4 time:2021-08-17T16:12:36.840+08:00&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T16:12:36.840+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:23138,   &quot;ctx&quot;:&quot;conn362334&quot;,&quot;msg&quot;:&quot;Shutting down&quot;,&quot;attr&quot;:{&quot;exitCode&quot;:0}}&lt;br/&gt;
2&#12289;startup using 4.4.8 time:2021-08-17T16:12:44.132+08:00&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T16:12:44.132+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;CONTROL&quot;,  &quot;id&quot;:20698,   &quot;ctx&quot;:&quot;main&quot;,&quot;msg&quot;:&quot;***** SERVER RESTARTED *****&quot;}&lt;br/&gt;
3&#12289;crash using 4.4.8 time:2021-08-18T10:07:19.822+08:00&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;from the following log:this duplicate _id is op:u,update lead to this duplicate&amp;#93;&lt;/span&gt;&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T10:07:19.821+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:21238,   &quot;ctx&quot;:&quot;ReplWriterWorker-1777&quot;,&quot;msg&quot;:&quot;Writer worker caught exception&quot;,&lt;/p&gt;

&lt;p&gt;3&#12289;the output of validate() on each affected node&lt;br/&gt;
i need to enter standalone so that i can execute validate.&lt;br/&gt;
db.runCommand( &lt;/p&gt;
{ validate: &quot;TrackerDetail&quot;, full: true }
&lt;p&gt; )&lt;/p&gt;

&lt;p&gt;&quot;advice&quot; : &quot;A corrupt namespace has been detected. See &lt;a href=&quot;http://dochub.mongodb.org/core/data-recovery&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://dochub.mongodb.org/core/data-recovery&lt;/a&gt; for recovery steps.&quot;,&lt;br/&gt;
why i shutdown it normal.it can corrupt.&lt;/p&gt;

&lt;p&gt;i  have upload three files to  support uploader location.&lt;/p&gt;

&lt;p&gt;curl -X POST &lt;a href=&quot;https://upload.box.com/api/2.0/files/content&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://upload.box.com/api/2.0/files/content&lt;/a&gt; -H &apos;Authorization: Bearer 1!uKuATr9f9QobvE9AHsQ_XR44g2Igm3uZzp0s_N91uYhMJ1sGZDLfivIg6zjWUjwF352nq59XPNE0eZyt53AE5fXpHUmfFxV5DeJQ4HItqE_1rMu9QbN6xIiwKhYo_caHj_xME3IB5iESmpg0V8X0KX5A94deYuyKGtZuViQQcmMahbsFm3r5FqapqwW6MfjUWZdmNAjmRpkmDBybnJ7PrN1mOcar7OUMXE0p1toQQ-7BWvUUtK2VzH9flsBoqA8E2Z_kvMxE_aUZhDrB4TImv1IGpzC6pGrCtjK8y-Mu6Fh7N89ozs2cJFGx6FryzuWIyDizzh3bp1ufE6PEm_ieOwLNgYHYRChNabeHyBwUEAY6xu5WEuZiOdJhUDHMW2DWAf-TTGqdFbpkR489Q1FnAy-oOEY.&apos; -H &apos;Content-Type: multipart/form-data&apos; -F attributes=&apos;{&quot;name&quot;: &quot;diagnostic.data.tar&quot;, &quot;parent&quot;: {&quot;id&quot;: &quot;143586341049&quot;}}&apos; -F file=@diagnostic.data.tar &amp;gt; /dev/null&lt;br/&gt;
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current&lt;br/&gt;
                                 Dload  Upload   Total   Spent    Left  Speed&lt;br/&gt;
100  192M  100  1171  100  192M     44  7503k  0:00:26  0:00:26 -&lt;del&gt;:&lt;/del&gt;&lt;del&gt;:&lt;/del&gt;-     0&lt;/p&gt;

&lt;p&gt;curl -X POST &lt;a href=&quot;https://upload.box.com/api/2.0/files/content&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://upload.box.com/api/2.0/files/content&lt;/a&gt; \&lt;br/&gt;
&amp;gt; -H &apos;Authorization: Bearer 1!uKuATr9f9QobvE9AHsQ_XR44g2Igm3uZzp0s_N91uYhMJ1sGZDLfivIg6zjWUjwF352nq59XPNE0eZyt53AE5fXpHUmfFxV5DeJQ4HItqE_1rMu9QbN6xIiwKhYo_caHj_xME3IB5iESmpg0V8X0KX5A94deYuyKGtZuViQQcmMahbsFm3r5FqapqwW6MfjUWZdmNAjmRpkmDBybnJ7PrN1mOcar7OUMXE0p1toQQ-7BWvUUtK2VzH9flsBoqA8E2Z_kvMxE_aUZhDrB4TImv1IGpzC6pGrCtjK8y-Mu6Fh7N89ozs2cJFGx6FryzuWIyDizzh3bp1ufE6PEm_ieOwLNgYHYRChNabeHyBwUEAY6xu5WEuZiOdJhUDHMW2DWAf-TTGqdFbpkR489Q1FnAy-oOEY.&apos; \&lt;br/&gt;
&amp;gt; -H &apos;Content-Type: multipart/form-data&apos; \&lt;br/&gt;
&amp;gt; -F attributes=&apos;{&quot;name&quot;: &quot;shard2.log&quot;, &quot;parent&quot;: {&quot;id&quot;: &quot;143586341049&quot;}}&apos; \&lt;br/&gt;
&amp;gt; -F file=@shard2.log &amp;gt; /dev/null&lt;br/&gt;
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current&lt;br/&gt;
                                 Dload  Upload   Total   Spent    Left  Speed&lt;br/&gt;
100 84.9M  100  1161  100 84.9M     12   922k  0:01:36  0:01:34  0:00:02     0&lt;/p&gt;

&lt;p&gt;curl -X POST &lt;a href=&quot;https://upload.box.com/api/2.0/files/content&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://upload.box.com/api/2.0/files/content&lt;/a&gt; \&lt;br/&gt;
&amp;gt; -H &apos;Authorization: Bearer 1!uKuATr9f9QobvE9AHsQ_XR44g2Igm3uZzp0s_N91uYhMJ1sGZDLfivIg6zjWUjwF352nq59XPNE0eZyt53AE5fXpHUmfFxV5DeJQ4HItqE_1rMu9QbN6xIiwKhYo_caHj_xME3IB5iESmpg0V8X0KX5A94deYuyKGtZuViQQcmMahbsFm3r5FqapqwW6MfjUWZdmNAjmRpkmDBybnJ7PrN1mOcar7OUMXE0p1toQQ-7BWvUUtK2VzH9flsBoqA8E2Z_kvMxE_aUZhDrB4TImv1IGpzC6pGrCtjK8y-Mu6Fh7N89ozs2cJFGx6FryzuWIyDizzh3bp1ufE6PEm_ieOwLNgYHYRChNabeHyBwUEAY6xu5WEuZiOdJhUDHMW2DWAf-TTGqdFbpkR489Q1FnAy-oOEY.&apos; \&lt;br/&gt;
&amp;gt; -H &apos;Content-Type: multipart/form-data&apos; \&lt;br/&gt;
&amp;gt; -F attributes=&apos;{&quot;name&quot;: &quot;validate.txt&quot;, &quot;parent&quot;: {&quot;id&quot;: &quot;143586341049&quot;}}&apos; \&lt;br/&gt;
&amp;gt; -F file=@validate.txt &amp;gt; /dev/null&lt;br/&gt;
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current&lt;br/&gt;
                                 Dload  Upload   Total   Spent    Left  Speed&lt;br/&gt;
100 37433  100  1160  100 36273    519  16253  0:00:02  0:00:02 -&lt;del&gt;:&lt;/del&gt;&lt;del&gt;:&lt;/del&gt;- 16258&lt;/p&gt;</comment>
                            <comment id="4005687" author="601290552@qq.com" created="Wed, 18 Aug 2021 02:25:06 +0000"  >&lt;p&gt;Hi Eric:&lt;br/&gt;
Slowly, all the seconday instances started reporting duplicate key errors, causing the instance to go down.so i guess that it can not  logical corruption for all secondays except primary.the instance is the small,Otherwise, no synchronization is completed, and only the primary instance is alive and eventually becomes read-only.&lt;/p&gt;</comment>
                            <comment id="4005655" author="601290552@qq.com" created="Wed, 18 Aug 2021 01:46:56 +0000"  >&lt;p&gt;Hi Eric:&lt;br/&gt;
   I will sort through the process for you. It needs some time.&lt;/p&gt;</comment>
                            <comment id="4005497" author="eric.sedor" created="Tue, 17 Aug 2021 22:28:33 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=601290552%40qq.com&quot; class=&quot;user-hover&quot; rel=&quot;601290552@qq.com&quot;&gt;601290552@qq.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;This error message leads us to suspect logical corruption. Please make a complete copy of each affected node&apos;s $dbpath directory to safeguard so that you can work off of the current $dbpath.&lt;/p&gt;

&lt;p&gt;Our ability to determine the source of this corruption depends greatly on your ability to provide:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The logs for the affected nodes, including before, leading up to, and after the first sign of corruption.&lt;/li&gt;
	&lt;li&gt;a clear timeline of activities (with timestamps and timezones) including server restarts, replica set changes, version changes, etc.&lt;/li&gt;
	&lt;li&gt;the output of &lt;a href=&quot;https://docs.mongodb.com/manual/reference/command/validate/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;validate()&lt;/a&gt; on each affected node&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Would you please archive (tar or zip) the validate output, mongod.log files, and the &lt;tt&gt;$dbpath/diagnostic.data&lt;/tt&gt; directory (the contents are described &lt;a href=&quot;https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/#full-time-diagnostic-data-capture&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;) and upload them to this &lt;a href=&quot;https://amphora.corp.mongodb.com/public/upload/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJmb2xkZXJfaWQiOiIxNDM1ODYzNDEwNDkiLCJleHAiOjE2MzE4Mjk3MzN9.LFvBuQ9W-TbZ5NtACx9LkNB-dgBHvedtxcWB5CxokdI&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;support uploader&lt;/a&gt; location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.&lt;/p&gt;

&lt;p&gt;Then, can you elaborate on the timeline of what has occurred here?&lt;/p&gt;

&lt;p&gt;The ideal resolution will likely be to perform a clean &lt;a href=&quot;https://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#automatically-sync-a-member&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;resync&lt;/a&gt; from an unaffected node. &lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Eric&lt;/p&gt;</comment>
                            <comment id="4005120" author="601290552@qq.com" created="Tue, 17 Aug 2021 19:51:37 +0000"  >&lt;p&gt;i check from primary:&lt;br/&gt;
this record create at 	&quot;createTime&quot; : ISODate(&quot;2021-08-14T03:03:00.557Z&quot;),&lt;br/&gt;
i find 	&quot;currentTime&quot; : ISODate(&quot;2021-08-17T17:40:25Z&quot;),&lt;br/&gt;
this time the same from secondary carsh time.so it is  maybe update at primary.&lt;br/&gt;
but i can not find this record on secondary.why it indicate duplicate key&lt;br/&gt;
	&quot;createTime&quot; : ISODate(&quot;2021-08-14T03:03:00.557Z&quot;),&lt;br/&gt;
	&quot;rBranch&quot; : &quot;769902&quot;,&lt;br/&gt;
	&quot;rRegion&quot; : &quot;755920&quot;,&lt;br/&gt;
	&quot;_class&quot; : &quot;cn.yto.wdgj.rw.pojo.mongodo.WantedMonitorNewDO&quot;,&lt;br/&gt;
	&quot;contractOrg&quot; : null,&lt;br/&gt;
	&quot;currentOrg&quot; : &quot;769902&quot;,&lt;br/&gt;
	&quot;currentTime&quot; : ISODate(&quot;2021-08-17T17:40:25Z&quot;),&lt;/p&gt;</comment>
                            <comment id="4005092" author="601290552@qq.com" created="Tue, 17 Aug 2021 19:43:22 +0000"  >&lt;p&gt;another seconday for shard3 again carsh:&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-18T03:40:38.987+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;F&quot;,  &quot;c&quot;:&quot;REPL&quot;,     &quot;id&quot;:21235,   &quot;ctx&quot;:&quot;initandlisten&quot;,&quot;msg&quot;:&quot;Failed to apply batch of operations&quot;,&quot;attr&quot;:{&quot;numOperationsInBatch&quot;:1013,&quot;firstOperation&quot;:{&quot;lsid&quot;:{&quot;id&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;e0556e1e-9897-41c1-a4dc-6afa6d9e50db&quot;}
&lt;p&gt;,&quot;uid&quot;:{&quot;$binary&quot;:&lt;/p&gt;
{&quot;base64&quot;:&quot;kafFGlFnFmnb4qY6Zo1wZcN0z6AuZ9x3brpDqqhzH/U=&quot;,&quot;subType&quot;:&quot;0&quot;}
&lt;p&gt;}},&quot;txnNumber&quot;:2339,&quot;op&quot;:&quot;u&quot;,&quot;ns&quot;:&quot;returnMonitor.exp_want_new_monitor&quot;,&quot;ui&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;0814628c-47c0-4210-a9d4-1c7c0a4161ee&quot;}
&lt;p&gt;,&quot;o&quot;:{&quot;$v&quot;:1,&quot;$set&quot;:{&quot;backFlag&quot;:1,&quot;currentOp&quot;:751,&quot;currentTime&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T17:43:19.000Z&quot;}
&lt;p&gt;,&quot;dealStatus&quot;:19,&quot;dealTime&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T17:43:19.207Z&quot;}
&lt;p&gt;,&quot;opList.7&quot;:162922219900751,&quot;orgBranch&quot;:&quot;579038&quot;}},&quot;o2&quot;:{&quot;_id&quot;:{&quot;$oid&quot;:&quot;6118aaf405f34e22a4f681a4&quot;}},&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1629222199,&quot;i&quot;:6}},&quot;t&quot;:6,&quot;v&quot;:2,&quot;wall&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T17:43:19.212Z&quot;}
&lt;p&gt;,&quot;stmtId&quot;:0,&quot;prevOpTime&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:0,&quot;i&quot;:0}},&quot;t&quot;:-1}},&quot;lastOperation&quot;:{&quot;lsid&quot;:{&quot;id&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;783598dd-63df-4489-9505-1ab1923be921&quot;}
&lt;p&gt;,&quot;uid&quot;:{&quot;$binary&quot;:&lt;/p&gt;
{&quot;base64&quot;:&quot;kafFGlFnFmnb4qY6Zo1wZcN0z6AuZ9x3brpDqqhzH/U=&quot;,&quot;subType&quot;:&quot;0&quot;}
&lt;p&gt;}},&quot;txnNumber&quot;:3065,&quot;op&quot;:&quot;u&quot;,&quot;ns&quot;:&quot;returnMonitor.exp_want_new_monitor&quot;,&quot;ui&quot;:&lt;/p&gt;
{&quot;$uuid&quot;:&quot;0814628c-47c0-4210-a9d4-1c7c0a4161ee&quot;}
&lt;p&gt;,&quot;o&quot;:{&quot;$v&quot;:1,&quot;$set&quot;:{&quot;currentOp&quot;:171,&quot;currentOrg&quot;:&quot;270902&quot;,&quot;currentTime&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T17:44:43.000Z&quot;}
&lt;p&gt;,&quot;nextOrg&quot;:&quot;712001&quot;,&quot;opList.5&quot;:162922228300171,&quot;warnLevel&quot;:20}},&quot;o2&quot;:{&quot;_id&quot;:{&quot;$oid&quot;:&quot;6119bda405f34e22a4f78351&quot;}},&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:1629222294,&quot;i&quot;:2}},&quot;t&quot;:6,&quot;v&quot;:2,&quot;wall&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T17:44:54.404Z&quot;}
&lt;p&gt;,&quot;stmtId&quot;:0,&quot;prevOpTime&quot;:{&quot;ts&quot;:{&quot;$timestamp&quot;:{&quot;t&quot;:0,&quot;i&quot;:0}},&quot;t&quot;:-1}},&quot;failedWriterThread&quot;:12,&quot;error&quot;:&quot;DuplicateKey{ keyPattern: &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;, keyValue: &lt;/p&gt;
{ _id: ObjectId(&apos;611732d899faf22fbb509820&apos;) }
&lt;p&gt; }: E11000 duplicate key error collection: returnMonitor.exp_want_new_monitor index: &lt;em&gt;id&lt;/em&gt; dup key: &lt;/p&gt;
{ _id: ObjectId(&apos;611732d899faf22fbb509820&apos;) }
&lt;p&gt;&quot;}}&lt;/p&gt;

&lt;p&gt;when i enter standalone execute find,database not find this record.&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; db.exp_want_new_monitor.find({_id: ObjectId(&apos;611732d899faf22fbb509820&apos;)})&lt;br/&gt;
&amp;gt; db.exp_want_new_monitor.count();&lt;br/&gt;
17806952&lt;/p&gt;</comment>
                            <comment id="4003635" author="601290552@qq.com" created="Tue, 17 Aug 2021 11:16:38 +0000"  >&lt;p&gt;when i initiate secondary,from log indicates&#65306;&lt;br/&gt;
{&quot;t&quot;:&lt;/p&gt;
{&quot;$date&quot;:&quot;2021-08-17T19:14:52.076+08:00&quot;}
&lt;p&gt;,&quot;s&quot;:&quot;I&quot;,  &quot;c&quot;:&quot;NETWORK&quot;,  &quot;id&quot;:22990,   &quot;ctx&quot;:&quot;conn245&quot;,&quot;msg&quot;:&quot;DBException handling request, closing client connection&quot;,&quot;attr&quot;:{&quot;error&quot;:&quot;NotWritablePrimary: Not-primary error while processing &apos;find&apos; operation  on &apos;returnMonitor&apos; database via fire-and-forget command execution.&quot;}}&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                                                <inwardlinks description="is caused by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1858595">WT-7995</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 17 Aug 2021 22:28:33 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        2 years, 16 weeks, 2 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            2 years, 16 weeks, 2 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eric.sedor@mongodb.com</customfieldvalue>
            <customfieldvalue>601290552@qq.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzy0xr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzi4l3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[eric.sedor@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzxn6n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>