[GODRIVER-1324] Can't insert 16 MiB documents Created: 30/Sep/19 Updated: 28/Oct/23 Resolved: 01/Oct/19 |
|
| Status: | Closed |
| Project: | Go Driver |
| Component/s: | Wire Protocol |
| Affects Version/s: | 1.1.1 |
| Fix Version/s: | 1.1.2 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Tim Fogarty | Assignee: | Divjot Arora (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Description |
|
Here is a gist of the program I used to attempt to insert a 16 MiB document: https://gist.github.com/tfogo/fea1a80c565f7e97f7464c6b0c39640f Running this results in this output:
This error is due to this line: This line is reducing the space available for documents in a message. The comment implies this is to make space for commands. But messages can be larger than 16 MiB (48 MB by default) so I believe it's incorrect to reserve space for commands that should be used for documents. I tested reducing reservedCommandBufferBytes to 0. This allows all documents up to 16 MiB in size to be inserted. However, trying to insert a document that is exactly 16 MiB causes another error:
This error comes from the server. It looks like the OP_MSG sent to the server could be malformed. |
| Comments |
| Comment by Githook User [ 01/Oct/19 ] |
|
Author: {'username': 'divjotarora', 'email': 'divjot.arora@10gen.com', 'name': 'Divjot Arora'}Message: Allow insertion of 16MiB documents. Change-Id: I739164acff37a0405c1623ab192aa5ab16962326 |
| Comment by Githook User [ 01/Oct/19 ] |
|
Author: {'name': 'Divjot Arora', 'username': 'divjotarora', 'email': 'divjot.arora@10gen.com'}Message: Allow insertion of 16MiB documents. Change-Id: I330894d6e288cf21de923c3d66aebe0491e5fd6a |
| Comment by Tim Fogarty [ 01/Oct/19 ] |
|
divjot.arora Nice! Looks like both issues are solved now. Thanks for solving it so quickly! ryan.chipman I'll revendor this into mongo-tools-common for MGOMIRROR-290. |
| Comment by Divjot Arora (Inactive) [ 30/Sep/19 ] |
|
CR: https://review.gerrithub.io/c/mongodb/mongo-go-driver/+/469914 |
| Comment by Jeffrey Yemin [ 30/Sep/19 ] |
|
This is not what we should be doing. The largest document that we allow to be inserted should be 16MB, and the command document should be allowed to be an additional 16K. This is complicated by document sequences, in that the insert command itself will be really small because it doesn't actually contain the documents array. In that case, we should allow each document in the type 1 payload document sequence to be 16MB. |
| Comment by Ryan Chipman [ 30/Sep/19 ] |
|
It looks like what the server allows is 16M + 16K: https://github.com/mongodb/mongo/blob/134a4083953270e8a11430395357fb70a29047ad/jstests/noPassthrough/query_knobs_validation.js#L146 |
| Comment by Divjot Arora (Inactive) [ 30/Sep/19 ] |
|
I think I see the issue. In batches.go, we'll pass 16MiB as the target batch size and then reserve 16KB per reservedCommandBufferBytes, so the largest document possible is actually 16Mib - 16KB. The 16KB is reserved as overhead for the actual command document. jeff.yemin Is this not what we should be doing? |
| Comment by Ryan Chipman [ 30/Sep/19 ] |
|
thanks for jumping on this so quickly jeff.yemin and divjot.arora. Let us know if there's anything else we can do on our end to help out |
| Comment by Tim Fogarty [ 30/Sep/19 ] |
|
Thanks for the correction about the 48 MB. It should still be within 16 MB + 16 KB though. I'm targeting 4.0.3. |
| Comment by Divjot Arora (Inactive) [ 30/Sep/19 ] |
|
tim.fogarty What server version is this targeting? As Jeff said, the 48 MB only applies to OP_MSG, which is used on server versions 3.6 and above. For OP_MSG, insert.documents isn't a field that we send because the documents are sent as a a document sequence. |
| Comment by Tim Fogarty [ 30/Sep/19 ] |
|
jeff.yemin in the fist case, the wiremessage doesn't even get constructed. The error happens before that can occur. If I set reservedCommandBufferBytes to 0, when I attempt to insert a 16 MiB document the wire message is 101 bytes long. (So there's definitely an issue constructing the message there.) If I try to insert a document that is one byte smaller than 16 MiB, the message is 16777331 bytes long. This is 16 MiB + 115 B. Which looks correct. |
| Comment by Jeffrey Yemin [ 30/Sep/19 ] |
|
tim.fogarty can you ascertain how large the actual message is? I think drivers should allow for command documents up to 16MB + 16K, so I want to make sure the message is not larger than that. FYI, the 48MB max message size only applies when the driver is using OP_MSG type 1 payloads, which is only for the bulk write commands (insert, update, delete). Otherwise, the practical limit is still 16MB + 16K. |
| Comment by Tim Fogarty [ 30/Sep/19 ] |
|
We need to solve this to fix MGOMIRROR-290 |