Implementing buffer.h in terms of PyByteArray in order to avoid memory copy

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Performance
    • None
    • Python Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      Re-implement buffer.h in terms of PyByteArray in order to avoid the big memory copy at the end of BSON encoding. The C api methods would return bytearray instead of bytes.

      Spike gist: https://gist.github.com/blink1073/3735f9428ce29faca19ab1d901f05529
      Spike draft PR: https://github.com/blink1073/mongo-python-driver/pull/12

      TLDR; using PyByteArray saves 1× the document size in peak memory usage, while giving a modest performance increase.

      Definition of done

      Implement the change and verify there are no C correctness issues. Verify on a perf patch build that there are no perf regressions.

      Pitfalls

      Correctness and performance.

            Assignee:
            Unassigned
            Reporter:
            Steve Silvester
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: