-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Rust Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Context
The Python team is building an experimental Rust-backed BSON codec (PYTHON-6207). We want it to decode BSON using the bson crate's raw module rather than a hand-rolled parser, so that BSON input handling is validated by the bson create instead of being duplicated in PyMongo. It's C decoder has had defects in exactly this area (bounded-scan fixes under PYTHON-6110 and PYTHON-6111).
One PyMongo decoder requirement cannot be expressed with the raw module today. PyMongo lets applications choose how invalid UTF-8 in keys and string values is handled: CodecOptions.unicode_decode_error_handler accepts strict, replace, ignore, surrogateescape, or any registered Python codec handler. Supporting that requires the original bytes, but the raw module only offers validated &str or a fixed lossy conversion:
- Iteration rejects invalid UTF-8 keys before the caller sees them: RawIter returns a UTF-8 error and the whole iteration stops.
- String-like values (strings, symbols, JavaScript code, code with scope, DBPointer namespaces) are exposed only as validated &str. The lossy alternative (value_utf8_lossy) always applies the replacement character and copies into an owned String. That is one specific policy, not a caller-chosen one.
Definition of done
- Keys can be read as raw bytes during iteration, so the caller decides how to decode them. For example: a key_bytes() accessor on RawElement, or an iteration mode that defers UTF-8 validation.
- The payload of string-like values can be read as raw bytes under the caller's decoding policy. For example: byte accessors on the RawBsonRef string variants, or a public RawValue accessor exposing the element kind, payload bytes, and source offset.
- The crate gains no pymongo-specific behavior: it hands over bytes, and the caller applies the policy.
The Python team is happy to contribute the change once the API direction is agreed.
Pitfalls
- This is public API surface. Validation timing changes: today iteration fails on the first invalid-UTF-8 key; deferring validation moves that error to value access or removes it, and existing callers' error handling changes with it.
- CStr guarantees valid UTF-8, and as_str() relies on that guarantee without re-checking. Byte access must not create a path that produces an unchecked &str.
- RawValue and its source offset are crate-private today (the offset is behind the facet feature). Exposing either is a deliberate API commitment, not an implementation leak.
- The owned lossy conversion and a borrowed bytes view serve different decoding policies; keep them distinct rather than merging them.