Do not persist large stream buffers

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Unknown
    • 2.1.0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Not Needed
    • None
    • C Drivers
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Proposal: When a client is pushed back onto the pool, reduce the size of large connection buffers.

      Background & Motivation

      HELP-78501 reports observed memory growth in a long-lived process. This appears to be due to reading large responses on a buffered stream. The long-lived buffered stream does not later reduce the size until the pool is destroyed.

      An example creating many cursors reading large responses reproduces the issue. Here is a simplified snippet:

      mongoc_client_t *client[100];
      for (int i = 0; i < 100; i++) {
          client[i] = mongoc_client_pool_pop (pool);
      }
      
      for (int i = 0; i < 100; i++) {
          // Server replies may be 16MB each.
          find_all_documents (client[i], "db", "big_collection");
      }
      
      // 1600 MB allocated and will not be freed until `mongoc_client_pool_destroy`.
      

      macOS Instruments shows many 16MB buffers allocated after calls to find_all_documents complete:

      After calling mongoc_client_pool_destroy, memory usage lowers to near 0.

      The default stream initiator creates buffered streams.

              Assignee:
              Kevin Albertson
              Reporter:
              Kevin Albertson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: