[CSHARP-2207] Large bulk writes not always splitting correctly Created: 06/Mar/18  Updated: 28/Oct/23  Resolved: 14/Mar/18

Status: Closed
Project: C# Driver
Component/s: Write Operations
Affects Version/s: 2.5
Fix Version/s: 2.5.1

Type: Bug Priority: Major - P3
Reporter: Ivan Fioravanti Assignee: Robert Stam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

MongoDB 3.6.3
MongoDB c# Driver 2.5.0


Issue Links:
Duplicate
is duplicated by CSHARP-2228 Size must be between 0 and 16793600(1... Closed
Related
related to SERVER-13061 Raise write batch limit to 100,000 op... Closed
Case:

 Description   

From: https://docs.mongodb.com/manual/reference/limits/

Write Command Batch Limit Size

100,000 writes are allowed in a single batch operation, defined by a single request to the server.

Changed in version 3.6: The limit raises from 1,000 to 100,000 writes. This limit also applies to legacy OP_INSERT messages.

We are using MongoDB C# driver 2.5.

This change is causing a lot of issues like:

  • InsertMany is now failing when large batch are sent to MongoDB 3.6.3.
  • WriteBulkAsync is failing when many WriteModels are sent to MongoDB 3.6.3
    Error is Command insert failed: BSONObj size: 16793671 (0x1004047) is invalid. Size must be between 0 and 16793600(16MB)

We had to change code on our side to split manually InserMany operation in multiple batch and same for bulk operations, but we can't be sure that size will not be over 16MB. I think this should be handled by Driver and moreover it should be possible to change this limit on server settings side in order to let customer being able to tune it accordingly to specific usage.



 Comments   
Comment by Robert Stam [ 29/Mar/18 ]

Very soon. We are fixing a small number of remaining issues targeted for 2.5.1 and then we will release it.

Comment by Sagar Thakur [ 28/Mar/18 ]

any idea when 2.5.1 will be out?

Comment by Githook User [ 16/Mar/18 ]

Author:

{'email': 'robert@robertstam.org', 'name': 'rstam', 'username': 'rstam'}

Message: CSHARP-2207: Large bulk writes not always splitting correctly
Branch: v2.5.x
https://github.com/mongodb/mongo-csharp-driver/commit/5ba8320b0e678b7cdfe5e54fd1fed0d951b6c4d2

Comment by Githook User [ 14/Mar/18 ]

Author:

{'email': 'robert@robertstam.org', 'name': 'rstam', 'username': 'rstam'}

Message: CSHARP-2207: Large bulk writes not always splitting correctly
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/6b1b509101f61eda95652de7194cfdbdc742f303

Comment by Robert Stam [ 07/Mar/18 ]

I can confirm that this is a C# driver bug, so the ticket was moved from SERVER-33688 to CSHARP-2207.

It can be reproduced using:

var client = new MongoClient("mongodb://localhost");
var database = client.GetDatabase("test");
var collection = database.GetCollection<BsonDocument>("test");
database.DropCollection("test");
 
var documents = new BsonDocument[]
{
    new BsonDocument { { "_id", 1 }, { "filler", new string('x', 10000000) } },
    new BsonDocument { { "_id", 2 }, { "filler", new string('x', 6793527) } },
    new BsonDocument { { "_id", 3} }
};
collection.InsertMany(documents);

Generated at Wed Feb 07 21:41:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.