ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Rust Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      The Python team is building an experimental Rust-backed BSON codec (PYTHON-6207). We want it to decode BSON using the bson crate's raw module rather than a hand-rolled parser, so that BSON input handling is validated by the bson create instead of being duplicated in PyMongo. It's C decoder has had defects in exactly this area (bounded-scan fixes under PYTHON-6110 and PYTHON-6111).

      One PyMongo decoder requirement cannot be expressed with the raw module today. PyMongo lets applications choose how invalid UTF-8 in keys and string values is handled: CodecOptions.unicode_decode_error_handler accepts strict, replace, ignore, surrogateescape, or any registered Python codec handler. Supporting that requires the original bytes, but the raw module only offers validated &str or a fixed lossy conversion:

      • Iteration rejects invalid UTF-8 keys before the caller sees them: RawIter returns a UTF-8 error and the whole iteration stops.
      • String-like values (strings, symbols, JavaScript code, code with scope, DBPointer namespaces) are exposed only as validated &str. The lossy alternative (value_utf8_lossy) always applies the replacement character and copies into an owned String. That is one specific policy, not a caller-chosen one.

      Definition of done

      • Keys can be read as raw bytes during iteration, so the caller decides how to decode them. For example: a key_bytes() accessor on RawElement, or an iteration mode that defers UTF-8 validation.
      • The payload of string-like values can be read as raw bytes under the caller's decoding policy. For example: byte accessors on the RawBsonRef string variants, or a public RawValue accessor exposing the element kind, payload bytes, and source offset.
      • The crate gains no pymongo-specific behavior: it hands over bytes, and the caller applies the policy.

      The Python team is happy to contribute the change once the API direction is agreed.

      Pitfalls

      • This is public API surface. Validation timing changes: today iteration fails on the first invalid-UTF-8 key; deferring validation moves that error to value access or removes it, and existing callers' error handling changes with it.
      • CStr guarantees valid UTF-8, and as_str() relies on that guarantee without re-checking. Byte access must not create a path that produces an unchecked &str.
      • RawValue and its source offset are crate-private today (the offset is behind the facet feature). Exposing either is a deliberate API commitment, not an implementation leak.
      • The owned lossy conversion and a borrowed bytes view serve different decoding policies; keep them distinct rather than merging them.

            Assignee:
            Isabel Atkinson
            Reporter:
            Steve Silvester
            None
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: