Uploaded image for project: 'Go Driver'
  1. Go Driver
  2. GODRIVER-2469

NewInsufficientBytesError should not create a stack trace

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.10.0
    • Affects Version/s: 1.9.0
    • Component/s: BSON
    • None

      Summary

      NewInsufficientBytesError creates a stack trace, which allocates a lot of memory. For some reason, this is called a lot during normal operations, particularly by the Elements function.

      When ADL does format conversions, such as to Parquet, we recursively traverse documents and have observed trace memory allocations significantly greater than the underlying data traversed. This may be related to OOM problems we've seen in production. See attached pprof image.

      Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

      1.9.0

      How to Reproduce

      Traverse a large amount of BSON documents with Elements and look at memory allocations with the profiler.

      Additional Background

      The stack trace appears to have been introduced in the original codebase. The function to retrieve it, ErrorStack, does not appear to be used in the codebase. Because bsoncore is under bsonx, it should be safe to remove the stack trace functionality as it doesn't affect external, stable APIs.

        1. 2022-06-23_10-21.png
          350 kB
          David Golden

            Assignee:
            matt.dale@mongodb.com Matt Dale
            Reporter:
            david.golden@mongodb.com David Golden
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: