[CSHARP-725] InsertBatch throws duplicate key exception with too much data... (140.000 item, 13MB) but inserts all data correctly Created: 12/Apr/13 Updated: 20/Mar/14 Resolved: 29/Apr/13 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | None |
| Affects Version/s: | 1.8 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Serdar Büyüktemiz | Assignee: | Robert Stam |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
fully updated windows 7 development machine |
||
| Issue Links: |
|
||||||||
| Description |
|
In the code i first fill the items to List. then call the InsertBatch method. this throws these exceptions. if I look at the collection i see all the data inserted correctly. When I try with 10k data it works correctly. The data is clear in all lines. It is probably the size problem but the exception is miss leadling. Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. WriteConcern detected an error 'E11000 duplicate key error index: MYDB.MYDATA.$id dup key: { : ObjectId('51681daf1e72b21b1839877b') }'. (Response was { "err" : "E11000 duplicate key error index: MYDB.MYDATA.$id dup key: { : ObjectId('51681daf1e72b21b1839877b') }", "code" : 11000, "n" : 0, "connectionId" : 19, "ok" : 1.0 }). |
| Comments |
| Comment by Robert Stam [ 17/Apr/13 ] |
|
I answered the StackOverflow question. Correct, in my experience the performance difference between batch sizes of 1,000 documents per batch and 1,000,000 per batch will be very small. You could measure it to verify it. |
| Comment by Serdar Büyüktemiz [ 17/Apr/13 ] |
|
you are right i was using version 1.8.0 thank you very much. i am not re-trying in my application logic. also it would be nice if you can write an answer to this question? did i understand right? |
| Comment by Robert Stam [ 15/Apr/13 ] |
|
I see two issues reported above: 1. "message len 875442996 is too large 875442996" in the server logs The first issue seems like the same issue reported by https://jira.mongodb.org/browse/CSHARP-715 One of the characteristics of Can you double check that you are using v1.8.1 of the driver? It seems odd that you would be hitting this since it is supposedly fixed. The second issue might or might not be related to the first. Do you do any retries in your application? I'm wondering if you might be catching an exception thrown by the first issue and calling InsertBatch again to retry the operation. It would be fine if you want to call InsertBatch with smaller batches. That could be one way of working around this issue. Note that the main reason for calling InsertBatch instead of Insert is performance , and once your batch size reaches sizes even as small as a few hundred you should be getting 99% or more of the possible performance gains, so really large batch sizes don't really give any additional performance. |
| Comment by Serdar Büyüktemiz [ 15/Apr/13 ] |
|
I am leaving the _id empty and letting the driver generate them here are the db logs... Mon Apr 15 13:53:06 [conn3] recv(): message len 875442996 is too large875442996 i would like to remind again... another question for the bulk inserts log details... Mon Apr 15 13:52:38 [conn2] insert MYDB.MYDATA2 keyUpdates:0 locks(micros) w:368032 368ms Mon Apr 15 13:52:41 [conn3] build index done. scanned 0 total records. 0.001 secs Mon Apr 15 13:52:42 [conn3] build index done. scanned 0 total records. 0.001 secs Mon Apr 15 13:52:43 [conn3] build index done. scanned 0 total records. 0.001 secs Mon Apr 15 13:53:03 [conn3] build index done. scanned 0 total records. 0 secs |
| Comment by Robert Stam [ 12/Apr/13 ] |
|
It looks like you simply have a duplicate _id value in your MyData array. How are your _id values getting set? Do you set them yourself when you create the 144,000 MyData values? Or are you leaving the _id empty and letting the driver generate them? Are there any server side log file entries that coincide with this? |