[GODRIVER-1324] Can't insert 16 MiB documents Created: 30/Sep/19  Updated: 28/Oct/23  Resolved: 01/Oct/19

Status: Closed
Project: Go Driver
Component/s: Wire Protocol
Affects Version/s: 1.1.1
Fix Version/s: 1.1.2

Type: Bug Priority: Critical - P2
Reporter: Tim Fogarty Assignee: Divjot Arora (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to GODRIVER-2936 Bulk inserting many documents causes ... Closed

 Description   

Here is a gist of the program I used to attempt to insert a 16 MiB document: https://gist.github.com/tfogo/fea1a80c565f7e97f7464c6b0c39640f

Running this results in this output:

ERROR: an inserted document is too large 

This error is due to this line:

https://github.com/mongodb/mongo-go-driver/blob/1aaf676c48ebb378f02f7d93509d5f229c0cede2/x/mongo/driver/batches.go#L11

This line is reducing the space available for documents in a message. The comment implies this is to make space for commands. But messages can be larger than 16 MiB (48 MB by default) so I believe it's incorrect to reserve space for commands that should be used for documents.

I tested reducing reservedCommandBufferBytes to 0. This allows all documents up to 16 MiB in size to be inserted. However, trying to insert a document that is exactly 16 MiB causes another error:

ERROR: (Location40414) BSON field 'insert.documents' is missing but a required field 

This error comes from the server. It looks like the OP_MSG sent to the server could be malformed.



 Comments   
Comment by Githook User [ 01/Oct/19 ]

Author:

{'username': 'divjotarora', 'email': 'divjot.arora@10gen.com', 'name': 'Divjot Arora'}

Message: Allow insertion of 16MiB documents.

GODRIVER-1324

Change-Id: I739164acff37a0405c1623ab192aa5ab16962326
Branch: release/1.1
https://github.com/mongodb/mongo-go-driver/commit/af8e6c2695ec6f127505094f01dfd88a876dedbe

Comment by Githook User [ 01/Oct/19 ]

Author:

{'name': 'Divjot Arora', 'username': 'divjotarora', 'email': 'divjot.arora@10gen.com'}

Message: Allow insertion of 16MiB documents.

GODRIVER-1324

Change-Id: I330894d6e288cf21de923c3d66aebe0491e5fd6a
Branch: master
https://github.com/mongodb/mongo-go-driver/commit/7490fa5f89ee53bf642ba1969494987bc39a9918

Comment by Tim Fogarty [ 01/Oct/19 ]

divjot.arora Nice! Looks like both issues are solved now. Thanks for solving it so quickly!

ryan.chipman I'll revendor this into mongo-tools-common for MGOMIRROR-290.

Comment by Divjot Arora (Inactive) [ 30/Sep/19 ]

CR: https://review.gerrithub.io/c/mongodb/mongo-go-driver/+/469914

Comment by Jeffrey Yemin [ 30/Sep/19 ]

This is not what we should be doing. The largest document that we allow to be inserted should be 16MB, and the command document should be allowed to be an additional 16K. This is complicated by document sequences, in that the insert command itself will be really small because it doesn't actually contain the documents array. In that case, we should allow each document in the type 1 payload document sequence to be 16MB.

Comment by Ryan Chipman [ 30/Sep/19 ]

It looks like what the server allows is 16M + 16K: https://github.com/mongodb/mongo/blob/134a4083953270e8a11430395357fb70a29047ad/jstests/noPassthrough/query_knobs_validation.js#L146

Comment by Divjot Arora (Inactive) [ 30/Sep/19 ]

I think I see the issue. In batches.go, we'll pass 16MiB as the target batch size and then reserve 16KB per reservedCommandBufferBytes, so the largest document possible is actually 16Mib - 16KB. The 16KB is reserved as overhead for the actual command document. jeff.yemin Is this not what we should be doing?

Comment by Ryan Chipman [ 30/Sep/19 ]

thanks for jumping on this so quickly jeff.yemin and divjot.arora. Let us know if there's anything else we can do on our end to help out

Comment by Tim Fogarty [ 30/Sep/19 ]

Thanks for the correction about the 48 MB. It should still be within 16 MB + 16 KB though. I'm targeting 4.0.3.

Comment by Divjot Arora (Inactive) [ 30/Sep/19 ]

tim.fogarty What server version is this targeting? As Jeff said, the 48 MB only applies to OP_MSG, which is used on server versions 3.6 and above. For OP_MSG, insert.documents isn't a field that we send because the documents are sent as a a document sequence.

Comment by Tim Fogarty [ 30/Sep/19 ]

jeff.yemin in the fist case, the wiremessage doesn't even get constructed. The error happens before that can occur.

If I set reservedCommandBufferBytes to 0, when I attempt to insert a 16 MiB document the wire message is 101 bytes long. (So there's definitely an issue constructing the message there.)

If I try to insert a document that is one byte smaller than 16 MiB, the message is 16777331 bytes long. This is 16 MiB + 115 B. Which looks correct.

Comment by Jeffrey Yemin [ 30/Sep/19 ]

tim.fogarty can you ascertain how large the actual message is? I think drivers should allow for command documents up to 16MB + 16K, so I want to make sure the message is not larger than that.

FYI, the 48MB max message size only applies when the driver is using OP_MSG type 1 payloads, which is only for the bulk write commands (insert, update, delete). Otherwise, the practical limit is still 16MB + 16K.

Comment by Tim Fogarty [ 30/Sep/19 ]

We need to solve this to fix MGOMIRROR-290

Generated at Thu Feb 08 08:36:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.