[CDRIVER-2250] Using bson that contains UTF-8 strings with null bytes fails Created: 24/Aug/17  Updated: 28/Oct/23  Resolved: 25/Aug/17

Status: Closed
Project: C Driver
Component/s: libbson, libmongoc
Affects Version/s: 1.6.3, 1.7.0
Fix Version/s: 1.8.0

Type: Bug Priority: Major - P3
Reporter: Thijs Cadier Assignee: Hannes Magnusson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We ran into this when writing a Rust wrapper around libmongoc: https://github.com/thijsc/mongo-rust-driver

We use the bson Rust library to prepare bson byte arrays that get converted to libmongoc bsons using bson_new_from_data. This happens here: https://github.com/thijsc/mongo-rust-driver/blob/master/src/bsonc.rs#L31

Rust strings in the bson are converted to bson strings, these can contain null bytes as is allowed in the bson spec. When you try to insert these documents with either a normal insert or a bulk operation the insert fails with this error: "replacement document contains invalid keys)".

After debugging this it looks like libmongoc sets the allow_null option to false when validating bson. I've found no way to override this. It seems like we should just be able to use this bson byte array, since it conforms to the spec. How could we call normal inserts and bulk operations in such a way they will allow this bson?



 Comments   
Comment by Githook User [ 25/Aug/17 ]

Author:

{'email': 'bjori@php.net', 'name': 'Hannes Magnusson', 'username': 'bjori'}

Message: CDRIVER-2250 nModified is not available in MongoDB 2.4
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/7d24183ab5794b51a156911ea9cb49cfd887c5bb

Comment by Githook User [ 25/Aug/17 ]

Author:

{'email': 'bjori@php.net', 'name': 'Hannes Magnusson', 'username': 'bjori'}

Message: Merge branch 'CDRIVER-2250-null-value-in-strings'

Comment by Githook User [ 25/Aug/17 ]

Author:

{'email': 'bjori@php.net', 'name': 'Hannes Magnusson', 'username': 'bjori'}

Message: Merge branch 'CDRIVER-2250-null-value-in-strings'

Comment by Githook User [ 25/Aug/17 ]

Author:

{'email': 'bjori@php.net', 'name': 'Hannes Magnusson', 'username': 'bjori'}

Message: Merge branch 'CDRIVER-2250-null-value-in-strings'

Comment by Githook User [ 25/Aug/17 ]

Author:

{'email': 'bjori@php.net', 'name': 'Hannes Magnusson', 'username': 'bjori'}

Message: CDRIVER-2250 Cannot insert strings with null bytes
Branch: master
https://github.com/mongodb/mongo-c-driver/commit/cee8946363eb9286eea6b9191350343965cdaadd

Comment by Hannes Magnusson [ 24/Aug/17 ]

You are right. The validation thats done on inserts is missing BSON_VALIDATE_UTF_ALLOW_NULL

Comment by Thijs Cadier [ 24/Aug/17 ]

There's a test for this here: https://github.com/thijsc/mongo-rust-driver/pull/32/files#diff-5e55f18430efd105430896e3da14fdc1L77

If I fully disable bson validation it passed, if I enable it it fails. For bulk operations I haven't gotten it to work at all.

Comment by Thijs Cadier [ 24/Aug/17 ]

The field name contain no nulls, only the values.

Comment by A. Jesse Jiryu Davis [ 24/Aug/17 ]

The field names include NULLs, or only the values? Field names can't include NULL, although bsonspec.org merely implies that restriction instead of stating it explicitly.

Generated at Wed Feb 07 21:14:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.