-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Atlas Streams
-
Fully Compatible
-
None
-
3
-
TBD
-
None
-
None
-
None
-
None
-
None
-
None
-
None
We should create a thread executor and pass it into the AWS SDK ClientConfiguration struct to control the way we process PutObjectAsync requests.
Currently, according to the AWS SDK doc strings, it appears that we spawn a thread per call to PutObjectAsync and that thread probably holds onto the file that we're trying to upload until the callback function is called.
We can likely do a bit better than this by making a thread executor that:
limits the number of threads spawned by PutObjectAsync in a thread poolthrow away the file ASAP to reclaim memoryWe should confirm if the callback retains the copy of the file in the lambda or not. If we do hold onto it, then we should do this.
We should make a feature flag with different values for SPI tier. This feature flag value should control the number of max concurrent PutObjectAsync requests