[CDRIVER-26] Unable to work with large files in GridFS Created: 21/Nov/10  Updated: 19/Oct/16  Resolved: 17/Feb/11

Status: Closed
Project: C Driver
Component/s: None
Affects Version/s: 0.2
Fix Version/s: 0.2

Type: Bug Priority: Critical - P2
Reporter: Doug Mayer Assignee: Mathias Stearn
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Sticking a ~3GB file in the database (length of 2996783104 bytes) will break the C driver. I'm rusty on my C, but it appears everywhere we load "length" we use bson_iterator_int when it should probably be bson_iterator_double:

bson_find(&it, gfile->meta, "length");
length = bson_iterator_int(&it);

A full example is gridfile_get_numchunks: http://github.com/ctriolo/mongo-c-driver/blob/master/src/gridfs.c#L500-516

I would offer to help dig in and supply a patch, but before I do so I'd like someone with a better understanding of the driver to take a look and see if that is in fact the likely culprit.



 Comments   
Comment by Kyle Banker [ 17/Feb/11 ]

Doug,

gridfs_store_file can now store extra large (e.g., 3GB) files.

There's also a new API for incrementally writing buffers to a GridFS file. You can use this to write arbitrarily-sized files.

https://github.com/mongodb/mongo-c-driver/blob/master/test/gridfs.c#L149-154

This isn't set in stone yet, so let me know if you have any suggestions for the API.

Kyle

Comment by auto [ 17/Feb/11 ]

Author:

{u'login': u'banker', u'name': u'Kyle Banker', u'email': u'kylebanker@gmail.com'}

Message: CDRIVER-26 better GridFS buffer-by-buffer API
https://github.com/mongodb/mongo-c-driver/commit/863b07648447a79984579c4d7d365cd3d9d7b55a

Comment by auto [ 14/Feb/11 ]

Author:

{u'login': u'banker', u'name': u'Kyle Banker', u'email': u'kylebanker@gmail.com'}

Message: CDRIVER-26 work with large (> 3 GB) files;
Ability to write GridFS file buffer by buffer, quasi-streaming
https://github.com/mongodb/mongo-c-driver/commit/692b85eaaa210f6e421675b2f8b49221d613e9f4

Generated at Wed Feb 07 21:08:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.