<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:36:10 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[GODRIVER-1324] Can&apos;t insert 16 MiB documents</title>
                <link>https://jira.mongodb.org/browse/GODRIVER-1324</link>
                <project id="14289" key="GODRIVER">Go Driver</project>
                    <description>&lt;p&gt;Here is a gist of the program I used to attempt to insert a 16 MiB document:&#160;&lt;a href=&quot;https://gist.github.com/tfogo/fea1a80c565f7e97f7464c6b0c39640f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/tfogo/fea1a80c565f7e97f7464c6b0c39640f&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Running this results in this output:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ERROR: an inserted document is too large &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;This error is due to this line:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo-go-driver/blob/1aaf676c48ebb378f02f7d93509d5f229c0cede2/x/mongo/driver/batches.go#L11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-go-driver/blob/1aaf676c48ebb378f02f7d93509d5f229c0cede2/x/mongo/driver/batches.go#L11&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This line is reducing the space available for documents in a message. The comment implies this is to make space for commands. But messages can be larger than 16 MiB (48 MB by default) so I believe it&apos;s incorrect to reserve space for commands that should be used for documents.&lt;/p&gt;

&lt;p&gt;I tested reducing&#160;reservedCommandBufferBytes to 0. This allows all documents up to 16 MiB in size to be inserted. However, trying to insert a document that is exactly 16 MiB causes another error:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;ERROR: (Location40414) BSON field &lt;/span&gt;&lt;span style=&quot;color: blue; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;&apos;insert.documents&apos;&lt;/span&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt; is missing but a required field &lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;This error comes from the server. It looks like the OP_MSG sent to the server could be malformed.&lt;/p&gt;</description>
                <environment></environment>
        <key id="945933">GODRIVER-1324</key>
            <summary>Can&apos;t insert 16 MiB documents</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="2" iconUrl="https://jira.mongodb.org/images/icons/priorities/critical.svg">Critical - P2</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="divjot.arora@mongodb.com">Divjot Arora</assignee>
                                    <reporter username="tim.fogarty@mongodb.com">Tim Fogarty</reporter>
                        <labels>
                    </labels>
                <created>Mon, 30 Sep 2019 13:44:25 +0000</created>
                <updated>Sat, 28 Oct 2023 11:38:32 +0000</updated>
                            <resolved>Tue, 1 Oct 2019 14:34:38 +0000</resolved>
                                    <version>1.1.1</version>
                                    <fixVersion>1.1.2</fixVersion>
                                    <component>Wire Protocol</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="2442861" author="xgen-internal-githook" created="Tue, 1 Oct 2019 14:34:27 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;username&apos;: &apos;divjotarora&apos;, &apos;email&apos;: &apos;divjot.arora@10gen.com&apos;, &apos;name&apos;: &apos;Divjot Arora&apos;}
&lt;p&gt;Message: Allow insertion of 16MiB documents.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/GODRIVER-1324&quot; title=&quot;Can&amp;#39;t insert 16 MiB documents&quot; class=&quot;issue-link&quot; data-issue-key=&quot;GODRIVER-1324&quot;&gt;&lt;del&gt;GODRIVER-1324&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Change-Id: I739164acff37a0405c1623ab192aa5ab16962326&lt;br/&gt;
Branch: release/1.1&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-go-driver/commit/af8e6c2695ec6f127505094f01dfd88a876dedbe&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-go-driver/commit/af8e6c2695ec6f127505094f01dfd88a876dedbe&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2442659" author="xgen-internal-githook" created="Tue, 1 Oct 2019 13:29:21 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;Divjot Arora&apos;, &apos;username&apos;: &apos;divjotarora&apos;, &apos;email&apos;: &apos;divjot.arora@10gen.com&apos;}
&lt;p&gt;Message: Allow insertion of 16MiB documents.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/browse/GODRIVER-1324&quot; title=&quot;Can&amp;#39;t insert 16 MiB documents&quot; class=&quot;issue-link&quot; data-issue-key=&quot;GODRIVER-1324&quot;&gt;&lt;del&gt;GODRIVER-1324&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Change-Id: I330894d6e288cf21de923c3d66aebe0491e5fd6a&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo-go-driver/commit/7490fa5f89ee53bf642ba1969494987bc39a9918&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo-go-driver/commit/7490fa5f89ee53bf642ba1969494987bc39a9918&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2442305" author="tim.fogarty" created="Tue, 1 Oct 2019 10:14:05 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=divjot.arora&quot; class=&quot;user-hover&quot; rel=&quot;divjot.arora&quot;&gt;divjot.arora&lt;/a&gt; Nice! Looks like both issues are solved now. Thanks for solving it so quickly!&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ryan.chipman&quot; class=&quot;user-hover&quot; rel=&quot;ryan.chipman&quot;&gt;ryan.chipman&lt;/a&gt; I&apos;ll revendor this into mongo-tools-common for MGOMIRROR-290.&lt;/p&gt;</comment>
                            <comment id="2440964" author="divjot.arora" created="Mon, 30 Sep 2019 18:26:34 +0000"  >&lt;p&gt;CR:&#160;&lt;a href=&quot;https://review.gerrithub.io/c/mongodb/mongo-go-driver/+/469914&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.gerrithub.io/c/mongodb/mongo-go-driver/+/469914&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2440723" author="jeff.yemin" created="Mon, 30 Sep 2019 16:36:53 +0000"  >&lt;p&gt;This is not what we should be doing.  The largest document that we allow to be inserted should be 16MB, and the command document should be allowed to be an additional 16K.  This is complicated by document sequences, in that the insert command itself will be really small because it doesn&apos;t actually contain the &lt;tt&gt;documents&lt;/tt&gt; array.  In that case, we should allow each document in the type 1 payload document sequence to be 16MB.&lt;/p&gt;</comment>
                            <comment id="2440595" author="ryan.chipman" created="Mon, 30 Sep 2019 15:52:14 +0000"  >&lt;p&gt;It looks like what the server allows is &lt;tt&gt;16M + 16K&lt;/tt&gt;: &lt;a href=&quot;https://github.com/mongodb/mongo/blob/134a4083953270e8a11430395357fb70a29047ad/jstests/noPassthrough/query_knobs_validation.js#L146&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/blob/134a4083953270e8a11430395357fb70a29047ad/jstests/noPassthrough/query_knobs_validation.js#L146&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2440305" author="divjot.arora" created="Mon, 30 Sep 2019 14:51:11 +0000"  >&lt;p&gt;I think I see the issue. In &lt;tt&gt;batches.go&lt;/tt&gt;, we&apos;ll pass 16MiB as the target batch size and then reserve 16KB per &lt;tt&gt;reservedCommandBufferBytes&lt;/tt&gt;, so the largest document possible is actually 16Mib - 16KB. The 16KB is reserved as overhead for the actual command document. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jeff.yemin&quot; class=&quot;user-hover&quot; rel=&quot;jeff.yemin&quot;&gt;jeff.yemin&lt;/a&gt; Is this not what we should be doing?&lt;/p&gt;</comment>
                            <comment id="2440274" author="ryan.chipman" created="Mon, 30 Sep 2019 14:39:45 +0000"  >&lt;p&gt;thanks for jumping on this so quickly &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jeff.yemin&quot; class=&quot;user-hover&quot; rel=&quot;jeff.yemin&quot;&gt;jeff.yemin&lt;/a&gt; and &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=divjot.arora&quot; class=&quot;user-hover&quot; rel=&quot;divjot.arora&quot;&gt;divjot.arora&lt;/a&gt;. Let us know if there&apos;s anything else we can do on our end to help out&lt;/p&gt;</comment>
                            <comment id="2440250" author="tim.fogarty" created="Mon, 30 Sep 2019 14:30:40 +0000"  >&lt;p&gt;Thanks for the correction about the 48 MB. It should still be within 16 MB + 16 KB though. I&apos;m targeting 4.0.3.&lt;/p&gt;</comment>
                            <comment id="2440243" author="divjot.arora" created="Mon, 30 Sep 2019 14:27:25 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tim.fogarty&quot; class=&quot;user-hover&quot; rel=&quot;tim.fogarty&quot;&gt;tim.fogarty&lt;/a&gt; What server version is this targeting? As Jeff said, the 48 MB only applies to OP_MSG, which is used on server versions 3.6 and above. For OP_MSG, &lt;tt&gt;insert.documents&lt;/tt&gt; isn&apos;t a field that we send because the documents are sent as a a document sequence.&lt;/p&gt;</comment>
                            <comment id="2440146" author="tim.fogarty" created="Mon, 30 Sep 2019 14:09:36 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jeff.yemin&quot; class=&quot;user-hover&quot; rel=&quot;jeff.yemin&quot;&gt;jeff.yemin&lt;/a&gt; in the fist case, the wiremessage doesn&apos;t even get constructed. The error happens before that can occur.&lt;/p&gt;

&lt;p&gt;If I set&#160;reservedCommandBufferBytes to 0, when I attempt to insert a 16 MiB document the wire message is 101 bytes long. (So there&apos;s definitely an issue constructing the message there.)&lt;/p&gt;

&lt;p&gt;If I try to insert a document that is one byte smaller than 16 MiB, the message is&#160;16777331 bytes long. This is 16 MiB + 115 B. Which looks correct.&lt;/p&gt;</comment>
                            <comment id="2440065" author="jeff.yemin" created="Mon, 30 Sep 2019 13:50:26 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tim.fogarty&quot; class=&quot;user-hover&quot; rel=&quot;tim.fogarty&quot;&gt;tim.fogarty&lt;/a&gt; can you ascertain how large the actual message is? I think drivers should allow for command documents up to 16MB + 16K, so I want to make sure the message is not larger than that.&lt;/p&gt;

&lt;p&gt;FYI, the 48MB max message size only applies when the driver is using OP_MSG type 1 payloads, which is only for the bulk write commands (insert, update, delete).  Otherwise, the practical limit is still 16MB + 16K.&lt;/p&gt;</comment>
                            <comment id="2440050" author="tim.fogarty" created="Mon, 30 Sep 2019 13:45:32 +0000"  >&lt;p&gt;We need to solve this to fix MGOMIRROR-290&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2390145">GODRIVER-2936</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hvi0tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>