[CDRIVER-927] libbson streaming writer docs are unclear, perhaps misleading Created: 13/Oct/15  Updated: 11/Jan/16  Resolved: 19/Oct/15

Status: Closed
Project: C Driver
Component/s: docs, libbson
Affects Version/s: None
Fix Version/s: 1.3.0-beta0

Type: Bug Priority: Major - P3
Reporter: Tony Cebzanov Assignee: A. Jesse Jiryu Davis
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I'm trying to use libbson's streaming interface to write a sequence of records as described in this StackOverflow question. The streaming docs say the following:

Libbson provides mechanisms to stream BSON documents to and from file-descriptors. This can be convenient if you want to do stream oriented processing.

I do see that the reader interface, via bson_reader_new_from_file, does allow for reading from a file, but I see no similar interface for writing to a file as the docs suggest.

My code sample in the SO question shows some working code for writing from the bson_writer_t's buffer, but when dealing with an arbitrary number of sub-documents, this would mean I'd have to use one writer for the entire parent document, meaning the buffer can grow arbitrarily large and undermine the whole point of streaming.

Are there plans to add file writing support directly to the bson_writer interface as the docs seem to indicate, and if so, would this kind of incremental writing of documents be supported? This seems like a basic requirement for anyone who wants to emit larger documents with nested sub-documents.



 Comments   
Comment by Githook User [ 11/Jan/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: Merge branch 'master' into r1.2

Comment by Githook User [ 11/Jan/16 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-927 can't stream BSON to an fd
Branch: 1.3.0-dev
https://github.com/mongodb/libbson/commit/83f756f722f11cc7ac85dab328f3336e0ba5fdd5

Comment by Githook User [ 22/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: Merge branch 'master' into r1.2

Comment by Githook User [ 20/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: Merge branch 'master' into r1.2

Comment by Githook User [ 20/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-927 can't stream BSON to an fd
Branch: debian
https://github.com/mongodb/libbson/commit/83f756f722f11cc7ac85dab328f3336e0ba5fdd5

Comment by Githook User [ 20/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: Merge branch 'master' into r1.2

Comment by Githook User [ 20/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-927 can't stream BSON to an fd
Branch: r1.2
https://github.com/mongodb/libbson/commit/83f756f722f11cc7ac85dab328f3336e0ba5fdd5

Comment by Tony Cebzanov [ 19/Oct/15 ]

Roger that. Thanks.

Comment by A. Jesse Jiryu Davis [ 19/Oct/15 ]

I definitely see your argument, but all features compete with all other features for our limited attention. I don't think this will be a priority any time soon.

Comment by Tony Cebzanov [ 19/Oct/15 ]

OK, but I thought the intent was to promote libbson as a utility library for working with BSON data outside of MongoDB as well as within the server environment, in which case a streaming feature like this would be useful for third party developers who wish to read/write BSON. It's understandable that use cases for the MongoDB environment would be prioritized, but with BSON being promoted as an alternative to protobuf and friends, I thought there might be some interest in handling more generalized use cases that don't necessarily correspond to the needs of MongoDB.

Comment by Githook User [ 19/Oct/15 ]

Author:

{u'username': u'ajdavis', u'name': u'A. Jesse Jiryu Davis', u'email': u'jesse@mongodb.com'}

Message: CDRIVER-927 can't stream BSON to an fd
Branch: master
https://github.com/mongodb/libbson/commit/83f756f722f11cc7ac85dab328f3336e0ba5fdd5

Comment by A. Jesse Jiryu Davis [ 19/Oct/15 ]

Thanks for the report, we should update that line in the documentation.

There are not plans for incrementally writing large BSON documents to an fd. Since the server caps doc sizes at 16MB, I don't think there's much use in supporting handling documents larger than that in any other part of the driver. For example, streaming a huge document to an fd is a feature we could theoretically support; but a huge document can't be read or written to MongoDB so it doesn't seem worth the effort to make the driver better able to handle such a document.

Comment by Tony Cebzanov [ 13/Oct/15 ]

this would mean I'd have to use one reader for the entire parent document

Uh, make that "one writer for the entire parent document"

Generated at Wed Feb 07 21:11:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.