Memory leak when raising InvalidDocument with C extensions

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Unknown
    • 4.15.3
    • Affects Version/s: 4.11
    • Component/s: None
    • None
    • None
    • Python Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Running this script while tracking the process memory (with htop or another tool) shows a memory leak when raising InvalidDocument with C extensions.

      from bson import encode
      from bson.errors import InvalidDocument
      
      def main():
          small_str = "x" * 1024
          while True:
              for _ in range (10000):
                  doc = {"data": small_str, "invalid": set()}
                  try:
                      encode(doc)
                      raise Exception("Shouldn't get here")
                  except InvalidDocument as exc:
                      pass
                      # assert exc.document == doc
      
      if __name__ == "__main__":
          main()
      

      The leak increases rapidly which indicates to me that we're leaking the exception message or instance added in PYTHON-1982. In fact, when I checkout the commit before PYTHON-1982, there is no leak.

      The leak is in handle_invalid_doc_error here:
      https://github.com/mongodb/mongo-python-driver/blob/266caf02c4edb295eeafd1d7fcba0c8a52d2e611/bson/_cbsonmodule.c#L1650-L1690

            Assignee:
            Noah Stapp
            Reporter:
            Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: