[SERVER-6543] Possible to bulk insert more than 16MB of docs but not possible to retry via WBL Created: 20/Jul/12  Updated: 11/Jul/16  Resolved: 25/Oct/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 2.3.1

Type: Bug Priority: Major - P3
Reporter: Greg Studer Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: continueOnError, getLastError
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-6802 Do not enable ContinueOnError for all... Closed
Related
Operating System: ALL
Participants:

 Description   

The sharded insert code now correctly groups bulk inserts for different chunks, but the message size limit is larger than the bson size limit when retrying the bulk inserts via a WBL BSON object.

We can limit the bulk insert size we send to mongod to 1/2 the max BSON object size, or some other factor, however there's still the edge case when inserting documents very close to 16MB. We can still overrun the actual buffer size in this case due to the extra WBL metadata.



 Comments   
Comment by Greg Studer [ 21/Nov/12 ]

After some discussion, the full fix here (depending on SERVER-6802) is not backportable. The correct bulk-insert behavior for 2.0 and 2.2 is still to use application logic after the bulk insert to verify the insert was successful, or to use several individual inserts in batches followed by GLE, which has the same semantics and also requires application-level verification. However, there should be no hangs caused by badly sized bulk inserts.

Comment by Greg Studer [ 25/Oct/12 ]

Discussing backportability now.

Comment by Greg Studer [ 20/Sep/12 ]

Linked to SERVER-6802 - issue is related in that to correctly break up chunks into 8MB slices, we need to also ensure correct ContinueOnError semantics. Working on a fix now, ideally backportable.

Comment by Greg Studer [ 10/Aug/12 ]

patch in review

Comment by Greg Studer [ 20/Jul/12 ]

Second idea - maybe break up writebacks for bulk inserts into several sub-writebacks, avoiding the edge case. Have to be careful this is correct, however.

Generated at Thu Feb 08 03:12:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.