<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 08:37:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[GODRIVER-2142] unexpected EOF - large binary file (&gt;1mb  and &lt;10mb)</title>
                <link>https://jira.mongodb.org/browse/GODRIVER-2142</link>
                <project id="14289" key="GODRIVER">Go Driver</project>
                    <description>&lt;p&gt;Replicate this error -&lt;/p&gt;

&lt;p&gt;Create a zip file containing multiple files of different sizes which can come up to 10mb.&lt;/p&gt;

&lt;p&gt;Store that file as in a document&lt;/p&gt;

&lt;p&gt;something like:&lt;/p&gt;

&lt;p&gt;name: &quot;hello&quot; - string&lt;/p&gt;

&lt;p&gt;lifetime: 8943 - int&lt;/p&gt;

&lt;p&gt;config_file: Binary(&apos;...&apos;, 0) - zip is stored as binary format&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;After uploading this and then using golang to marshal it into a struct:&lt;/p&gt;

&lt;p&gt;by using a base64 decoder and then using ioutil.ReadAll this successfully decodes and returns a list of bytes (the zip file represented in bytes).&#160;&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;Issue:&lt;/p&gt;

&lt;p&gt;Running this multiple times I&apos;ve noticed it would fetch and decode successfully multiple times and rarely it would decode and report &quot;unexpected EOF&quot; error.&lt;/p&gt;

&lt;p&gt;I&apos;ve run my own mongo instances used a json file that adds the document to the collection and then used golang to read the document.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;The json file is not changing it remains the same. Most of the time it works fine, and sometimes it fails.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1865395">GODRIVER-2142</key>
            <summary>unexpected EOF - large binary file (&gt;1mb  and &lt;10mb)</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="benji.rewis@mongodb.com">Benji Rewis</assignee>
                                    <reporter username="pnaran@cryptoquantique.com">punit naran</reporter>
                        <labels>
                            <label>Bug</label>
                    </labels>
                <created>Wed, 1 Sep 2021 15:41:10 +0000</created>
                <updated>Fri, 27 Oct 2023 20:01:15 +0000</updated>
                            <resolved>Mon, 27 Sep 2021 12:00:32 +0000</resolved>
                                                                    <component>BSON</component>
                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="4086000" author="dbeng-pm-bot" created="Mon, 27 Sep 2021 12:00:34 +0000"  >&lt;p&gt;There hasn&apos;t been any recent activity on this ticket, so we&apos;re resolving it. Thanks for reaching out! Please feel free to comment on this if you&apos;re able to provide more information.&lt;/p&gt;</comment>
                            <comment id="4049970" author="benji.rewis" created="Fri, 10 Sep 2021 15:01:15 +0000"  >&lt;p&gt;Great thank you, &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=pnaran%40cryptoquantique.com&quot; class=&quot;user-hover&quot; rel=&quot;pnaran@cryptoquantique.com&quot;&gt;pnaran@cryptoquantique.com&lt;/a&gt;!&lt;/p&gt;

&lt;p&gt;I&#8217;m still a little confused about your use-case. I&#8217;ve attached some Go code that is a cleaned up version of the sample code you&#8217;ve provided. I&#8217;m assuming your code is similar to the one I&#8217;ve attached, and you&#8217;re getting an occasional unexpected EOF from the &lt;tt&gt;ioutil.ReadAll&lt;/tt&gt; call on L36.&lt;/p&gt;

&lt;p&gt;My main question is: what does your data look like in your collection? From what I can tell, you seem to be storing your file in a collection like this:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;bson.D {&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	{&#8220;name&#8221;, &#8220;myFile&#8221;},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	{&#8220;lifetime&#8221;, 8943},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	{&#8220;data_file&#8221;, bson.D{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		{&#8220;$binary&#8221;, bson.D{&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;			{&#8220;base64&#8221;, [base 64 encoding of zip file]},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;		},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;	},&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;}&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Is that correct?&lt;/p&gt;

&lt;p&gt;You have three cases in your provided code. One where &lt;tt&gt;data_file&lt;/tt&gt; is a &lt;tt&gt;map&lt;span class=&quot;error&quot;&gt;&amp;#91;string&amp;#93;&lt;/span&gt;map&lt;span class=&quot;error&quot;&gt;&amp;#91;string&amp;#93;&lt;/span&gt;string&lt;/tt&gt; (as is the case above), one where &lt;tt&gt;data_file&lt;/tt&gt; is a plain &lt;tt&gt;string&lt;/tt&gt;, and one where neither is true (in which case you error). &lt;/p&gt;

&lt;p&gt;The first case makes sense to me, and calling &lt;tt&gt;DecodeString&lt;/tt&gt; on &lt;span class=&quot;error&quot;&gt;&amp;#91;base 64 encoding of zip file&amp;#93;&lt;/span&gt; would certainly return the zip file represented in bytes. The second case, where &lt;tt&gt;data_file&lt;/tt&gt; is a plain &lt;tt&gt;string&lt;/tt&gt;, is confusing to me. What is the value of &lt;tt&gt;data_file&lt;/tt&gt; here? It looks like you&#8217;re treating the &lt;tt&gt;string&lt;/tt&gt; value as a list of bytes and encoding it to a base64 string. But, then you call &lt;tt&gt;ioutil.ReadAll&lt;/tt&gt; on that string, which is not possible, as the argument to &lt;tt&gt;ReadAll&lt;/tt&gt; must implement the &lt;a href=&quot;https://pkg.go.dev/io#Reader&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;io.Reader&lt;/a&gt; interface. Are you sure you&#8217;re calling &lt;tt&gt;ioutil.ReadAll(sEnc)&lt;/tt&gt;?&lt;/p&gt;

&lt;p&gt;As an aside, if you&#8217;re storing larger, whole files with MongoDB, I would recommend using the &lt;a href=&quot;https://docs.mongodb.com/manual/core/gridfs/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;GridFS feature&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="4046705" author="JIRAUSER1262186" created="Thu, 9 Sep 2021 08:55:59 +0000"  >&lt;ol&gt;
	&lt;li&gt;go-mongo 1.3.3&lt;/li&gt;
	&lt;li&gt;Yes, I had isolated some tests, I was first storing a JSON file as a collection to the db and pulling it down and reading it by marshalling it into a struct (the binary is of an interface and later base64 decoded and then asserted to []bytes by using ioutill.ReadAll)&lt;/li&gt;
	&lt;li&gt;For reading:&lt;br/&gt;
 (Similar code) &lt;br/&gt;
 type myData struct {&lt;br/&gt;
 &#160;&#160; DataFile interface{} `bson:&quot;data_file&quot;`&lt;br/&gt;
 }&lt;br/&gt;
 var mydata myData&lt;br/&gt;
 err := collections.FindOne(ctx, filter).Decode(&amp;amp;mydata)&lt;br/&gt;
 if err != nil { &#160;&#160; log.Println(err) &#160;&#160; return nil, fmt.Errorf(&quot;unable to find deployment config for data file&quot;) }&lt;br/&gt;
switch v := compFile.(type) {&lt;br/&gt;
 case map&lt;span class=&quot;error&quot;&gt;&amp;#91;string&amp;#93;&lt;/span&gt;map&lt;span class=&quot;error&quot;&gt;&amp;#91;string&amp;#93;&lt;/span&gt;string:&lt;br/&gt;
 &#160; deployConf, err := base64.RawStdEncoding.DecodeString(v&lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;$binary&amp;quot;&amp;#93;&lt;/span&gt;&lt;span class=&quot;error&quot;&gt;&amp;#91;&amp;quot;base64&amp;quot;&amp;#93;&lt;/span&gt;)&lt;br/&gt;
 &#160; ... handle error and return&lt;br/&gt;
 case string:&lt;br/&gt;
 &#160; sEnc := base64.StdEncoding.EncodeToString([]byte(mydata.DataFile))&lt;br/&gt;
 &#160; deployConf, err := ioutil.ReadAll(sEnc)&lt;br/&gt;
 &#160; ... handle error and return&lt;br/&gt;
 default:&lt;br/&gt;
 &#160;&#160; log.Fatalln(fmt.Errorf(&quot;Invalid data retrieved&quot;))&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="4045132" author="benji.rewis" created="Wed, 8 Sep 2021 15:40:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=pnaran%40cryptoquantique.com&quot; class=&quot;user-hover&quot; rel=&quot;pnaran@cryptoquantique.com&quot;&gt;pnaran@cryptoquantique.com&lt;/a&gt;&#160;apologies for the delay, and thanks for your bug report! We&apos;re investigating now.&lt;/p&gt;

&lt;p&gt;Some preliminary questions:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;What version of the Go driver are you using?&lt;/li&gt;
	&lt;li&gt;What function is causing the &lt;tt&gt;unexpected EOF&lt;/tt&gt; error? &lt;tt&gt;ioutil.ReadAll&lt;/tt&gt;?&lt;/li&gt;
	&lt;li&gt;Could you include the exact code you&apos;re using to read and then marshal the binary file?&lt;/li&gt;
&lt;/ol&gt;
</comment>
                    </comments>
                    <attachments>
                            <attachment id="334355" name="godriver2142_test.go" size="1341" author="benji.rewis@mongodb.com" created="Fri, 10 Sep 2021 15:01:35 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hzk407:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            </customfields>
    </item>
</channel>
</rss>