Limit the number of concurrent S3 $emit PutObjectAsync requests

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Atlas Streams
    • Fully Compatible
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      We should create a thread executor and pass it into the AWS SDK ClientConfiguration struct to control the way we process PutObjectAsync requests.

       

      Currently, according to the AWS SDK doc strings, it appears that we spawn a thread per call to PutObjectAsync and that thread probably holds onto the file that we're trying to upload until the callback function is called.

       

      We can likely do a bit better than this by making a thread executor that:

       

      1. limits the number of threads spawned by PutObjectAsync in a thread pool
      2. throw away the file ASAP to reclaim memory
        1. We should confirm if the callback retains the copy of the file in the lambda or not. If we do hold onto it, then we should do this.

       

      We should make a feature flag with different values for SPI tier. This feature flag value should control the number of max concurrent PutObjectAsync requests

              Assignee:
              Harendra Chawla
              Reporter:
              Andrew Chen
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: