[GODRIVER-2142] unexpected EOF - large binary file (>1mb and <10mb) Created: 01/Sep/21  Updated: 27/Oct/23  Resolved: 27/Sep/21

Status: Closed
Project: Go Driver
Component/s: BSON
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: punit naran Assignee: Benji Rewis (Inactive)
Resolution: Gone away Votes: 0
Labels: Bug
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File godriver2142_test.go    

 Description   

Replicate this error -

Create a zip file containing multiple files of different sizes which can come up to 10mb.

Store that file as in a document

something like:

name: "hello" - string

lifetime: 8943 - int

config_file: Binary('...', 0) - zip is stored as binary format

 

After uploading this and then using golang to marshal it into a struct:

by using a base64 decoder and then using ioutil.ReadAll this successfully decodes and returns a list of bytes (the zip file represented in bytes). 

 

Issue:

Running this multiple times I've noticed it would fetch and decode successfully multiple times and rarely it would decode and report "unexpected EOF" error.

I've run my own mongo instances used a json file that adds the document to the collection and then used golang to read the document.

 

The json file is not changing it remains the same. Most of the time it works fine, and sometimes it fails.



 Comments   
Comment by PM Bot [ 27/Sep/21 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information.

Comment by Benji Rewis (Inactive) [ 10/Sep/21 ]

Great thank you, pnaran@cryptoquantique.com!

I’m still a little confused about your use-case. I’ve attached some Go code that is a cleaned up version of the sample code you’ve provided. I’m assuming your code is similar to the one I’ve attached, and you’re getting an occasional unexpected EOF from the ioutil.ReadAll call on L36.

My main question is: what does your data look like in your collection? From what I can tell, you seem to be storing your file in a collection like this:

bson.D {
	{“name”, “myFile”},
	{“lifetime”, 8943},
	{“data_file”, bson.D{
		{“$binary”, bson.D{
			{“base64”, [base 64 encoding of zip file]},
		},
	},
}

Is that correct?

You have three cases in your provided code. One where data_file is a map[string]map[string]string (as is the case above), one where data_file is a plain string, and one where neither is true (in which case you error).

The first case makes sense to me, and calling DecodeString on [base 64 encoding of zip file] would certainly return the zip file represented in bytes. The second case, where data_file is a plain string, is confusing to me. What is the value of data_file here? It looks like you’re treating the string value as a list of bytes and encoding it to a base64 string. But, then you call ioutil.ReadAll on that string, which is not possible, as the argument to ReadAll must implement the io.Reader interface. Are you sure you’re calling ioutil.ReadAll(sEnc)?

As an aside, if you’re storing larger, whole files with MongoDB, I would recommend using the GridFS feature.

Comment by punit naran [ 09/Sep/21 ]
  1. go-mongo 1.3.3
  2. Yes, I had isolated some tests, I was first storing a JSON file as a collection to the db and pulling it down and reading it by marshalling it into a struct (the binary is of an interface and later base64 decoded and then asserted to []bytes by using ioutill.ReadAll)
  3. For reading:
    (Similar code)
    type myData struct {
       DataFile interface{} `bson:"data_file"`
    }
    var mydata myData
    err := collections.FindOne(ctx, filter).Decode(&mydata)
    if err != nil {    log.Println(err)    return nil, fmt.Errorf("unable to find deployment config for data file") }
    switch v := compFile.(type) {
    case map[string]map[string]string:
      deployConf, err := base64.RawStdEncoding.DecodeString(v["$binary"]["base64"])
      ... handle error and return
    case string:
      sEnc := base64.StdEncoding.EncodeToString([]byte(mydata.DataFile))
      deployConf, err := ioutil.ReadAll(sEnc)
      ... handle error and return
    default:
       log.Fatalln(fmt.Errorf("Invalid data retrieved"))

 

Comment by Benji Rewis (Inactive) [ 08/Sep/21 ]

pnaran@cryptoquantique.com apologies for the delay, and thanks for your bug report! We're investigating now.

Some preliminary questions:

  1. What version of the Go driver are you using?
  2. What function is causing the unexpected EOF error? ioutil.ReadAll?
  3. Could you include the exact code you're using to read and then marshal the binary file?
Generated at Thu Feb 08 08:37:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.