<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:28:09 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-12295] Migrations occassionally aborted due to duplicate key error on unique shard key</title>
                <link>https://jira.mongodb.org/browse/SERVER-12295</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello, we&apos;ve notice recently that the moveChunk directory will rapidly start using all of it&apos;s available disk space. Looking through the logs, we saw errors of Sun Jan  5 09:43:14 &lt;span class=&quot;error&quot;&gt;&amp;#91;migrateThread&amp;#93;&lt;/span&gt; ERROR: migrate failed: E11000 duplicate key error index: blah.$ownerid_1_docname_1_osvcname_1  dup key: { : &quot;11446719&quot;, : &quot;app.json&quot;, : &quot;blah&quot; }. We recently upgraded to 2.2.6 from 2.2.4 though I am not sure if that is relevant.&lt;/p&gt;

&lt;p&gt;When the disk fills, mongod crashes and fails over to the secondary. After clearing out the moveChunk dir, the partner (now primary node) will have the same behavior the next day. If we remove the offending duplicate key, the issue stops.&lt;/p&gt;</description>
                <environment>CentOS 6.4</environment>
        <key id="105204">SERVER-12295</key>
            <summary>Migrations occassionally aborted due to duplicate key error on unique shard key</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="thomas.rueckstiess@mongodb.com">Thomas Rueckstiess</assignee>
                                    <reporter username="dan.cooper@take2games.com">Dan Cooper</reporter>
                        <labels>
                    </labels>
                <created>Wed, 8 Jan 2014 19:37:10 +0000</created>
                <updated>Mon, 15 Nov 2021 22:49:49 +0000</updated>
                            <resolved>Tue, 10 Jun 2014 20:34:54 +0000</resolved>
                                    <version>2.2.6</version>
                                                    <component>Index Maintenance</component>
                    <component>Replication</component>
                    <component>Stability</component>
                    <component>Storage</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="616087" author="ramon.fernandez" created="Tue, 10 Jun 2014 20:32:47 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=al.gehrig%40rockstarsandiego.com&quot; class=&quot;user-hover&quot; rel=&quot;al.gehrig@rockstarsandiego.com&quot;&gt;al.gehrig@rockstarsandiego.com&lt;/a&gt;, &lt;/p&gt;

&lt;p&gt;we haven&apos;t heard back from you for some time, so I&apos;m going to mark this ticket as resolved. If you happen to run into this issue again feel free to re-open the ticket and provide additional information.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="570016" author="thomasr" created="Thu, 1 May 2014 22:42:52 +0000"  >&lt;p&gt;Okay, thanks for letting us know. We&apos;ll wait for you to provide additional details.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="570010" author="al.gehrig@rockstarsandiego.com" created="Thu, 1 May 2014 22:40:14 +0000"  >&lt;p&gt;Hi Thomas,&lt;/p&gt;

&lt;p&gt;Yes - we&apos;re still getting these from time-to-time. The next time we see one, I&apos;ll post to this ticket.&lt;/p&gt;

&lt;p&gt;-Al&lt;/p&gt;</comment>
                            <comment id="570002" author="thomasr" created="Thu, 1 May 2014 22:33:39 +0000"  >&lt;p&gt;Hi Al,&lt;/p&gt;

&lt;p&gt;I just wanted to follow up with you. Have you seen this issue appear again, or can we go ahead and resolve the ticket?&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="516003" author="asya" created="Fri, 14 Mar 2014 10:50:15 +0000"  >&lt;p&gt;Checking indexes now can help since they should be identical on each shard and if they are not it could suggest why this problem may be recurring.    If they &lt;b&gt;are&lt;/b&gt; identical it can rule out one possible source on the problem.&lt;/p&gt;

&lt;p&gt;I will put the ticket into waiting state, when this happens again, please let us know, attach related information and we will try to get to the bottom of it with you at that point.&lt;/p&gt;
</comment>
                            <comment id="515946" author="al.gehrig@rockstarsandiego.com" created="Fri, 14 Mar 2014 06:15:56 +0000"  >&lt;p&gt;We had to delete the document to prevent the storage from filling up. We will keep an eye out for this and the next time it happens we&apos;ll get the &quot;to&quot; and the &quot;from&quot; as well and the getIndexes().&lt;/p&gt;</comment>
                            <comment id="515941" author="asya" created="Fri, 14 Mar 2014 06:05:29 +0000"  >&lt;p&gt;Could you query for that document directly on the mongod in the &quot;from&quot; shard?&lt;/p&gt;

&lt;p&gt;It would also help to see db.collectionblah.getIndexes() on each shard&apos;s primary.&lt;/p&gt;</comment>
                            <comment id="515939" author="al.gehrig@rockstarsandiego.com" created="Fri, 14 Mar 2014 06:03:59 +0000"  >&lt;p&gt;Sorry - yes, that was a typo. I&apos;ve updated the comment.&lt;/p&gt;</comment>
                            <comment id="515937" author="asya" created="Fri, 14 Mar 2014 06:00:57 +0000"  >&lt;p&gt;If I&apos;m reading this right, it appears the error is from collection &quot;cloud_prod.members.yay.blah&quot; and the query on mongod was run against collection &quot;db.members.yay.iblah&quot;  (blah vs iblah) -  was this a typo when cutting-and-pasting the diagnostics?&lt;/p&gt;

</comment>
                            <comment id="515839" author="al.gehrig@rockstarsandiego.com" created="Fri, 14 Mar 2014 00:08:04 +0000"  >&lt;p&gt;We are still seeing this. We have seen incidents where there were actually duplicate keys which prevented chunk migration. But there have been other cases where there was not a duplicate key and chunk migration still failed. For example:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Mar  5 04:24:44 [repl writer worker 5] ERROR: writer worker caught exception: E11000 duplicate key error index: cloud_prod.members.yay.blah.$ownerid_1_docname_1_osvcname_1  dup key: { : &quot;35290103&quot;, : &quot;savedata.json&quot;, : &quot;sc&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;So I went and searched for that key&lt;/p&gt;

&lt;p&gt;Only one doc shows up when querying via the mongoD (this is the &quot;to&quot; mongoD):&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;rsgewrset12:PRIMARY&amp;gt; db.members.yay.blah.find({ ownerid: &quot;35290103&quot;, docname: &quot;savedata.json&quot;, osvcname: &quot;sc&quot; } )&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;52d0e3879c551e923a0b0575&quot;), &quot;data&quot; : { &quot;version&quot; : 1, &quot;ownerID&quot; ........&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;and only one doc shows up when querying via mongoS:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongos&amp;gt;  db.members.yay.blah.find({ ownerid: &quot;35290103&quot;, docname: &quot;savedata.json&quot;, osvcname: &quot;sc&quot; } )&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;52d0e3879c551e923a0b0575&quot;), &quot;data&quot; : { &quot;version&quot; : 1, &quot;ownerID&quot; ........&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;We end up with aborted chunk migrations in this case as well.&lt;/p&gt;</comment>
                            <comment id="515836" author="dan.cooper@take2games.com" created="Thu, 13 Mar 2014 23:57:56 +0000"  >&lt;p&gt;This is still an issue. A coworker will be adding details&lt;/p&gt;</comment>
                            <comment id="496022" author="david.hows" created="Mon, 10 Feb 2014 04:36:16 +0000"  >&lt;p&gt;Hi Dan,&lt;/p&gt;

&lt;p&gt;I&apos;m marking this issue as closed as we have not heard back from you.&lt;/p&gt;

&lt;p&gt;If there is further we can do for you on this issue please feel free to re-open it with details.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="480444" author="asya" created="Fri, 10 Jan 2014 21:33:21 +0000"  >&lt;p&gt;David,&lt;/p&gt;

&lt;p&gt;If I understand correctly, 10 is the &lt;em&gt;to&lt;/em&gt; shard and 27 is the from shard, correct?&lt;/p&gt;

&lt;p&gt;Would you run the same query and command on the primary of shard27 (from shard), which I believe is host rsgewrmng53?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;use cloud_prod&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.members.yay.car.find({ ownerid : &quot;16758570&quot;, docname : &quot;gamenp.json&quot;, osvcname : &quot;sc&quot; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.members.getIndexes()&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;If somehow the index on the &quot;from&quot; shard is not unique and there are duplicate records there it would explain why the duplicates keep showing up on the &quot;to&quot; shard - they are not already there before migration (which would clean them out anyway) they are being introduced by migration from the shard that may be missing a unique constraint on the index.&lt;/p&gt;</comment>
                            <comment id="479930" author="david.hows" created="Thu, 9 Jan 2014 23:43:05 +0000"  >&lt;p&gt;Thanks for those logs Dan,&lt;/p&gt;

&lt;p&gt;There is nothing in those logs to show why the document might have come back prior to the migration processing or explain why you received the duplicate key error during the migration.&lt;/p&gt;

&lt;p&gt;Would it be possible for you to run the same test again? But this time at a higher logging level (say log level 4)? I&apos;d like to see if this sheds any further light on what could be the problem. If you are able to do so, would you be able to delete the duplicate document, increase the logging on set27 and set10, then trigger the migration once again. Following this could you please upload the logs from both primaries of the two sets?&lt;/p&gt;

&lt;p&gt;This should give us significantly more information to see what might be causing these issues.&lt;/p&gt;

&lt;p&gt;Thanks for reporting this issue and your continuing patience,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="479420" author="dan.cooper@take2games.com" created="Thu, 9 Jan 2014 05:09:26 +0000"  >&lt;p&gt;So there was a case where there was a duplicate key error moving a chunk from set27 to set 10. Mng20 is the primary for that replica set.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:29 [conn579943] command admin.$cmd command: { moveChunk: &quot;cloud_prod.members.yay.car&quot;, from: &quot;ewrset27/ewrmng53:27017,ewrmng54:27017&quot;, to: &quot;ewrset10/ewrmng19:27017,ewrmng20:27017&quot;, fromShard: &quot;ewrset27&quot;, toShard: &quot;ewrset10&quot;, min: { ownerid: &quot;16723769&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, max: { ownerid: &quot;16792097&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, maxChunkSizeBytes: 67108864, shardId: &quot;cloud_prod.members.yay.car-ownerid_&quot;16723769&quot;docname_&quot;gamenp.json&quot;osvcname_&quot;sc&quot;&quot;, configdb: &quot;ewrmngcfg1:30001,ewrmngcfg2:30001,ewrmngcfg3:30001&quot;, secondaryThrottle: false } ntoreturn:1 keyUpdates:0 locks(micros) W:914 r:31419 reslen:693 3504ms&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; 
&lt;p&gt;I went and deleted the key from the primary mongoD.&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ewrset10:PRIMARY&amp;gt; db.members.yay.car.find({ ownerid : &quot;16758570&quot;, docname : &quot;gamenp.json&quot;, osvcname : &quot;sc&quot; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5238764b125a4204301d6944&quot;), &quot;data&quot; : { &quot;version&quot; : 1, &quot;ownerID&quot; : &quot;raaan90&quot;, &quot;sp0_order&quot; : { }, &quot;multiplayer0&quot; : { &quot;vehicle0&quot; : { &quot;45&quot; : &quot;5&quot;, &quot;46&quot; : &quot;203402306&quot;, &quot;47&quot; : &quot;67506433&quot;,&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&#8230;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&quot;appdata&quot; : { &quot;65&quot; : &quot;1.000000&quot;, &quot;66&quot; : &quot;1.000000&quot;, &quot;appUID&quot; : &quot;-1199345894&quot; } }, &quot;docname&quot; : &quot;gamenp.json&quot;, &quot;lastmodified&quot; : { &quot;DateTime&quot; : ISODate(&quot;2013-12-03T14:22:16.129Z&quot;), &quot;Ticks&quot; : NumberLong(&quot;635216773361295841&quot;) }, &quot;osvcname&quot; : &quot;sc&quot;, &quot;ownerid&quot; : &quot;16758570&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ewrset10:PRIMARY&amp;gt; db.members.yay.car.remove({ ownerid : &quot;16758570&quot;, docname : &quot;gamenp.json&quot;, osvcname : &quot;sc&quot; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ewrset10:PRIMARY&amp;gt; db.members.yay.car.find({ ownerid : &quot;16758570&quot;, docname : &quot;gamenp.json&quot;, osvcname : &quot;sc&quot; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; 
&lt;p&gt;and so it should be gone (no duplicate)&#8230; or so I thought&lt;/p&gt;

&lt;p&gt;Then chunk migration resumed and mongo then complains about it being back as a duplicate key&lt;/p&gt;
  &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:31 [conn579955] moveChunk request accepted at version 1142|3||5217f1a621ce2f768a688386&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:31 [conn579955] moveChunk number of documents: 6552&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:31 [initandlisten] connection accepted from 10.90.52.180:36457 #1141511 (4415 connections now open)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:32 [conn1141511] end connection 10.90.52.180:36457 (4414 connections now open)&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:32 [conn579955] moveChunk data transfer progress: { active: true, ns: &quot;cloud_prod.members.yay.car&quot;, from: &quot;ewrset27/ewrmng53:27017,ewrmng54:27017&quot;, min: { ownerid: &quot;16723769&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, max: { ownerid: &quot;16792097&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, shardKeyPattern: { ownerid: 1, docname: 1, osvcname: 1 }, state: &quot;clone&quot;, counts: { cloned: 1042, clonedBytes: 2055855, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:33 [conn579955] moveChunk data transfer progress: { active: false, ns: &quot;cloud_prod.members.yay.car&quot;, from: &quot;ewrset27/ewrmng53:27017,ewrmng54:27017&quot;, min: { ownerid: &quot;16723769&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, max: { ownerid: &quot;16792097&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, shardKeyPattern: { ownerid: 1, docname: 1, osvcname: 1 }, state: &quot;fail&quot;, errmsg: &quot;E11000 duplicate key error index: cloud_prod.members.yay.car.$ownerid_1_docname_1_osvcname_1  dup key: { : &quot;16758570&quot;, : &quot;gamenp.json&quot;, : &quot;sc&quot; }&quot;, counts: { cloned: 1042, clonedBytes: 2055855, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;Wed Jan  8 18:40:33 [conn579955] warning: moveChunk error transferring data caused migration abort: { active: false, ns: &quot;cloud_prod.members.yay.car&quot;, from: &quot;ewrset27/ewrmng53:27017,ewrmng54:27017&quot;, min: { ownerid: &quot;16723769&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, max: { ownerid: &quot;16792097&quot;, docname: &quot;gamenp.json&quot;, osvcname: &quot;sc&quot; }, shardKeyPattern: { ownerid: 1, docname: 1, osvcname: 1 }, state: &quot;fail&quot;, errmsg: &quot;E11000 duplicate key error index: cloud_prod.members.yay.car.$ownerid_1_docname_1_osvcname_1  dup key: { : &quot;16758570&quot;, : &quot;gamenp.json&quot;, : &quot;sc&quot; }&quot;, counts: { cloned: 1042, clonedBytes: 2055855, catchup: 0, steady: 0 }, ok: 1.0 }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;And sure enough it&#8217;s back on set 10:&lt;/p&gt;
  &lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ewrset10:PRIMARY&amp;gt; db.members.yay.car.find({ ownerid : &quot;16758570&quot;, docname : &quot;gamenp.json&quot;, osvcname : &quot;sc&quot; })&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;{ &quot;_id&quot; : ObjectId(&quot;5238764b125a4204301d6944&quot;), &quot;data&quot; : { &quot;version&quot; : 1, &quot;ownerID&quot; : &quot;raaan90&quot;, &quot;sp0_order&quot; : { }, &quot;multiplayer0&quot; : { &quot;vehicle0&quot; : { &quot;45&quot; : &quot;5&quot;, &quot;46&quot; : &quot;203402306&quot;, &quot;47&quot; : &quot;67506433&quot;, &quot;48&quot; : &quot;133505289&quot;, &quot;49&quot; : &quot;-2122317693&quot;, &quot;10&quot; : &quot;3&quot;, &quot;11&quot; : &quot;0&quot;, &quot;12&quot; : &quot;0&quot;, &quot;13&quot; : &quot;0&quot;, &quot;14&quot; : &quot;0&quot;, &quot;50&quot; : &quot;2118394178&quot;, &quot;15&quot; : &quot;4&quot;, &quot;51&quot; : &quot;116&quot;, &quot;16&quot; : &quot;2108042618&quot;, &quot;52&quot; : &quot;1.000000&quot;, &quot;17&quot; : &quot;1766676233&quot;, &quot;18&quot; : &quot;-1390777827&quot;, &quot;53&quot; : &quot;car&quot;, &quot;54&quot; : &quot;None&quot;, &quot;19&quot; : &quot;-1751761149&quot;, &quot;20&quot; : &quot;1732399718&quot;, &quot;25&quot; : &quot;0&quot;, &quot;26&quot; : &quot;255&quot;, &quot;27&quot; : &quot;255&quot;, &quot;28&quot; : &quot;255&quot;, &quot;29&quot; : &quot;0&quot;, &quot;69&quot; : &quot;7234&quot;, &quot;1073741823&quot;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&#8230;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}, &quot;mp0_order&quot; : { }, &quot;sp1_order&quot; : { }, &quot;appdata&quot; : { &quot;65&quot; : &quot;1.000000&quot;, &quot;66&quot; : &quot;1.000000&quot;, &quot;appUID&quot; : &quot;-1199345894&quot; } }, &quot;docname&quot; : &quot;gamenp.json&quot;, &quot;lastmodified&quot; : { &quot;DateTime&quot; : ISODate(&quot;2013-12-03T14:22:16.129Z&quot;), &quot;Ticks&quot; : NumberLong(&quot;635216773361295841&quot;) }, &quot;osvcname&quot; : &quot;sc&quot;, &quot;ownerid&quot; : &quot;16758570&quot; }&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;  &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Unless I delete it from mongoS, there doesn&#8217;t appear to be a way to get rid of the duplicate issue (and, of course, deleting it from mongoS means actually losing data).&lt;/p&gt;</comment>
                            <comment id="479409" author="david.hows" created="Thu, 9 Jan 2014 04:34:57 +0000"  >&lt;p&gt;Thanks Dan,&lt;/p&gt;

&lt;p&gt;We will wait for the logs and review then.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
David&lt;/p&gt;</comment>
                            <comment id="479328" author="dan.cooper@take2games.com" created="Wed, 8 Jan 2014 23:47:39 +0000"  >&lt;p&gt;We just tried to delete the key from the mongod, it tried to move the chunk and then complained about the duplicate again. I&apos;ll attach logs shortly.&lt;/p&gt;</comment>
                            <comment id="479327" author="eliot" created="Wed, 8 Jan 2014 23:45:57 +0000"  >&lt;p&gt;Can you send the full log where you got the duplicate key exception and the FROM side of that chunk migrate?&lt;/p&gt;

&lt;p&gt;Do you know if the document in question is being modified during migration?&lt;/p&gt;</comment>
                            <comment id="479320" author="dan.cooper@take2games.com" created="Wed, 8 Jan 2014 23:35:33 +0000"  >&lt;p&gt;Can you delete the document with the offending key on the TO side of the migrate? - Yes, but we&apos;d like to know why we keep getting duplicates though. It seems to happen randomly on any given shard.&lt;/p&gt;

&lt;p&gt;How many shards are there? - 40 shards. One primary, one secondary and an arbitor on each shard.&lt;/p&gt;

&lt;p&gt;Also, Is there a specific reason you haven&apos;t upgrade to 2.4 yet? - We&apos;re planning to soon, Last time we tried we had an issue with mongostat not working properly, which we use heavily. You guys weren&apos;t able to replicate the problem.&lt;/p&gt;</comment>
                            <comment id="479317" author="eliot" created="Wed, 8 Jan 2014 23:28:00 +0000"  >&lt;p&gt;Can you delete the document with the offending key on the TO side of the migrate?&lt;br/&gt;
How many shards are there?&lt;br/&gt;
Also, Is there a specific reason you haven&apos;t upgrade to 2.4 yet?&lt;/p&gt;</comment>
                            <comment id="479302" author="dan.cooper@take2games.com" created="Wed, 8 Jan 2014 23:12:10 +0000"  >&lt;p&gt;Shard Key is &quot;key&quot; : &lt;/p&gt;
{ &quot;ownerid&quot; : 1, &quot;docname&quot; : 1, &quot;osvcname&quot; : 1 }

&lt;p&gt;Indexes are:&lt;br/&gt;
 [&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;{&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;v&quot; : 1,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;key&quot; : &lt;/p&gt;
{
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;_id&quot; : 1
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;}
&lt;p&gt;,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;ns&quot; : &quot;cloud_prod.members.gta5.car&quot;,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;name&quot; : &quot;&lt;em&gt;id&lt;/em&gt;&quot;&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;},&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;{&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;v&quot; : 1,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;key&quot; : &lt;/p&gt;
{
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;ownerid&quot; : 1,
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;docname&quot; : 1,
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;osvcname&quot; : 1
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;}
&lt;p&gt;,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;unique&quot; : true,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;ns&quot; : &quot;cloud_prod.members.gta5.car&quot;,&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&quot;name&quot; : &quot;ownerid_1_docname_1_osvcname_1&quot;&lt;br/&gt;
&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160;}&lt;br/&gt;
]&lt;/p&gt;</comment>
                            <comment id="479293" author="eliot" created="Wed, 8 Jan 2014 22:54:45 +0000"  >&lt;p&gt;Some questions.&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;What&apos;s the shard key?&lt;/li&gt;
	&lt;li&gt;Can you send the indexes on the collection? (db.&amp;lt;collection&amp;gt;.getIndexes())&lt;/li&gt;
&lt;/ul&gt;
</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>22.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 8 Jan 2014 22:54:45 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        9 years, 36 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            9 years, 36 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>al.gehrig@rockstarsandiego.com</customfieldvalue>
            <customfieldvalue>asya.kamsky@mongodb.com</customfieldvalue>
            <customfieldvalue>dan.cooper@take2games.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>eliot</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>thomas.rueckstiess@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrm4bb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrv8uf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>96265</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsp0h3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>