<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Wed Feb 07 21:37:39 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[CSHARP-725] InsertBatch throws duplicate key exception with too much data... (140.000 item, 13MB) but inserts all data correctly</title>
                <link>https://jira.mongodb.org/browse/CSHARP-725</link>
                <project id="10041" key="CSHARP">C# Driver</project>
                    <description>&lt;p&gt;In the code i first fill the items to List.&lt;br/&gt;
var datas = new List&amp;lt;MyData&amp;gt;(); // 144000 MyData&lt;/p&gt;

&lt;p&gt;then call the InsertBatch method.&lt;br/&gt;
var _collection = _mongoDatabase.GetCollection&amp;lt;MyData&amp;gt;(&quot;MYDATA&quot;, WriteConcern.Acknowledged);&lt;br/&gt;
_collection.InsertBatch(datas);&lt;/p&gt;


&lt;p&gt;this throws these exceptions. if I look at the collection i see all the data inserted correctly. When I try with 10k data it works correctly. The data is clear in all lines. It is probably the size problem but the exception is miss leadling.&lt;/p&gt;


&lt;p&gt;Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.&lt;/p&gt;

&lt;p&gt;WriteConcern detected an error &apos;E11000 duplicate key error index: MYDB.MYDATA.$&lt;em&gt;id&lt;/em&gt;  dup key: { : ObjectId(&apos;51681daf1e72b21b1839877b&apos;) }&apos;. (Response was { &quot;err&quot; : &quot;E11000 duplicate key error index: MYDB.MYDATA.$&lt;em&gt;id&lt;/em&gt;  dup key: { : ObjectId(&apos;51681daf1e72b21b1839877b&apos;) }&quot;, &quot;code&quot; : 11000, &quot;n&quot; : 0, &quot;connectionId&quot; : 19, &quot;ok&quot; : 1.0 }).&lt;/p&gt;
</description>
                <environment>fully updated windows 7 development machine</environment>
        <key id="71629">CSHARP-725</key>
            <summary>InsertBatch throws duplicate key exception with too much data... (140.000 item, 13MB) but inserts all data correctly</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="robert@mongodb.com">Robert Stam</assignee>
                                    <reporter username="serdarb">Serdar B&#252;y&#252;ktemiz</reporter>
                        <labels>
                    </labels>
                <created>Fri, 12 Apr 2013 14:47:22 +0000</created>
                <updated>Thu, 20 Mar 2014 16:39:13 +0000</updated>
                            <resolved>Mon, 29 Apr 2013 15:16:25 +0000</resolved>
                                    <version>1.8</version>
                                                                        <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="315357" author="rstam" created="Wed, 17 Apr 2013 12:42:51 +0000"  >&lt;p&gt;I answered the StackOverflow question.&lt;/p&gt;

&lt;p&gt;Correct, in my experience the performance difference between batch sizes of 1,000 documents per batch and 1,000,000 per batch will be very small. You could measure it to verify it.&lt;/p&gt;</comment>
                            <comment id="315226" author="serdarb" created="Wed, 17 Apr 2013 07:57:52 +0000"  >&lt;p&gt;you are right i was using version 1.8.0 &lt;br/&gt;
i changed the version and it fixed.&lt;/p&gt;

&lt;p&gt;thank you very much.&lt;/p&gt;

&lt;p&gt;i am not re-trying in my application logic. &lt;br/&gt;
can it be the WriteConcern.Acknowledged causing that re-try?&lt;/p&gt;

&lt;p&gt;also it would be nice if you can write an answer to this question?&lt;br/&gt;
&lt;a href=&quot;http://stackoverflow.com/questions/16053197/what-is-the-optimum-bulk-item-count-with-insertbatch-method-in-mongodb-c-sharp-d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://stackoverflow.com/questions/16053197/what-is-the-optimum-bulk-item-count-with-insertbatch-method-in-mongodb-c-sharp-d&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;did i understand right?&lt;br/&gt;
that when batch inserting &lt;br/&gt;
100 * 1.000 docs vs 1 * 100.000 docs&lt;br/&gt;
has nearly no performance diffrence&lt;/p&gt;</comment>
                            <comment id="314076" author="rstam" created="Mon, 15 Apr 2013 22:32:05 +0000"  >&lt;p&gt;I see two issues reported above:&lt;/p&gt;

&lt;p&gt;1. &quot;message len 875442996 is too large 875442996&quot; in the server logs&lt;br/&gt;
2. WriteConcern detected an error &apos;E11000 duplicate key error&apos; exception thrown by driver&lt;/p&gt;

&lt;p&gt;The first issue seems like the same issue reported by &lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-715&quot; title=&quot;InsertBatch fails when large batch has to be split into smaller sub batches&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CSHARP-715&quot;&gt;&lt;del&gt;CSHARP-715&lt;/del&gt;&lt;/a&gt; (and fixed in 1.8.1):&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-715&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;https://jira.mongodb.org/browse/CSHARP-715&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One of the characteristics of &lt;a href=&quot;https://jira.mongodb.org/browse/CSHARP-715&quot; title=&quot;InsertBatch fails when large batch has to be split into smaller sub batches&quot; class=&quot;issue-link&quot; data-issue-key=&quot;CSHARP-715&quot;&gt;&lt;del&gt;CSHARP-715&lt;/del&gt;&lt;/a&gt; is that all the documents were in fact inserted even though InsertBatch threw an exception, which matches the behavior you are seeing.&lt;/p&gt;

&lt;p&gt;Can you double check that you are using v1.8.1 of the driver? It seems odd that you would be hitting this since it is supposedly fixed.&lt;/p&gt;

&lt;p&gt;The second issue might or might not be related to the first. Do you do any retries in your application? I&apos;m wondering if you might be catching an exception thrown by the first issue and calling InsertBatch again to retry the operation.&lt;/p&gt;

&lt;p&gt;It would be fine if you want to call InsertBatch with smaller batches. That could be one way of working around this issue. Note that the main reason for calling InsertBatch instead of Insert is performance , and once your batch size reaches sizes even as small as a few hundred you should be getting 99% or more of the possible performance gains, so really large batch sizes don&apos;t really give any additional performance.&lt;/p&gt;</comment>
                            <comment id="313586" author="serdarb" created="Mon, 15 Apr 2013 11:00:52 +0000"  >&lt;p&gt;I am leaving the _id empty and letting the driver generate them&lt;/p&gt;

&lt;p&gt;here are the db logs...&lt;/p&gt;

&lt;p&gt;Mon Apr 15 13:53:06 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; recv(): message len 875442996 is too large875442996&lt;/p&gt;

&lt;p&gt;i would like to remind again... &lt;br/&gt;
even it gives exceptions the item count is correct in the db...&lt;/p&gt;


&lt;p&gt;another question for the bulk inserts &lt;br/&gt;
should i devide them to 50k and insert them in a for loop ?&lt;/p&gt;




&lt;p&gt;log details...&lt;/p&gt;

&lt;p&gt;Mon Apr 15 13:52:38 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; insert MYDB.MYDATA2 keyUpdates:0 locks(micros) w:368032 368ms&lt;br/&gt;
Mon Apr 15 13:52:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; insert MYDB.MYDATA2 keyUpdates:0 locks(micros) w:360042 360ms&lt;br/&gt;
Mon Apr 15 13:52:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; insert MYDB.MYDATA2 keyUpdates:0 locks(micros) w:194648 195ms&lt;br/&gt;
Mon Apr 15 13:52:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; OpCounters::gotOp unknown op: 19376&lt;br/&gt;
Mon Apr 15 13:52:39 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; Assertion: 16141:cannot translate opcode 19376&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; ntdll.dll       RtlRestoreContext+0x2e2&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; mongod.exe      ???&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; kernel32.dll    BaseThreadInitThunk+0xd&lt;br/&gt;
Mon Apr 15 13:52:40 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn2&amp;#93;&lt;/span&gt; AssertionException handling request, closing client connection: 16141 cannot translate opcode 19376&lt;br/&gt;
Mon Apr 15 13:52:41 &lt;span class=&quot;error&quot;&gt;&amp;#91;initandlisten&amp;#93;&lt;/span&gt; connection accepted from 127.0.0.1:6142 #3 (2 connections now open)&lt;br/&gt;
Mon Apr 15 13:52:41 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index MYDB.X &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Apr 15 13:52:41 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0.001 secs&lt;br/&gt;
Mon Apr 15 13:52:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index MYDB.Y &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Apr 15 13:52:42 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0.001 secs&lt;br/&gt;
Mon Apr 15 13:52:43 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index MYDB.Z &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Apr 15 13:52:43 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0.001 secs&lt;br/&gt;
Mon Apr 15 13:52:58 &lt;span class=&quot;error&quot;&gt;&amp;#91;FileAllocator&amp;#93;&lt;/span&gt; allocating new datafile C:/mongodb/data/MYDB.2, filling with zeroes...&lt;br/&gt;
Mon Apr 15 13:52:59 &lt;span class=&quot;error&quot;&gt;&amp;#91;FileAllocator&amp;#93;&lt;/span&gt; done allocating datafile C:/mongodb/data/MYDB.2, size: 256MB,  took 0.43 secs&lt;br/&gt;
Mon Apr 15 13:53:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index MYDB.MYDATA &lt;/p&gt;
{ _id: 1 }
&lt;p&gt;Mon Apr 15 13:53:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; build index done.  scanned 0 total records. 0 secs&lt;br/&gt;
Mon Apr 15 13:53:03 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; insert MYDB.MYDATA keyUpdates:0 locks(micros) w:473516 473ms&lt;br/&gt;
Mon Apr 15 13:53:05 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; insert MYDB.MYDATA keyUpdates:0 locks(micros) w:475588 476ms&lt;br/&gt;
Mon Apr 15 13:53:06 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; insert MYDB.MYDATA keyUpdates:0 locks(micros) w:471620 471ms&lt;br/&gt;
Mon Apr 15 13:53:06 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; recv(): message len 875442996 is too large875442996&lt;br/&gt;
Mon Apr 15 13:53:06 &lt;span class=&quot;error&quot;&gt;&amp;#91;conn3&amp;#93;&lt;/span&gt; end connection 127.0.0.1:6142 (1 connection now open)&lt;/p&gt;</comment>
                            <comment id="311985" author="rstam" created="Fri, 12 Apr 2013 14:57:28 +0000"  >&lt;p&gt;It looks like you simply have a duplicate _id value in your MyData array.&lt;/p&gt;

&lt;p&gt;How are your _id values getting set? Do you set them yourself when you create the 144,000 MyData values? Or are you leaving the _id empty and letting the driver generate them?&lt;/p&gt;

&lt;p&gt;Are there any server side log file entries that coincide with this?&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="69932">CSHARP-715</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hrn8xr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>49391</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>