Investigate the single threaded performance of S3EmitOperator

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Gone away
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Determine the relative performance of emitting to local S3 compared to local mongo and local kafka.

       

      If the performance values are relatively similar, we should not implement the multi threaded approach to uploading S3 objects.

       

      We can wait for the parallelism effort to shape what the parallelism constructs should look like in the engine.

       

      If the performance is significantly worse, we should implement the worker pool architecture

              Assignee:
              Unassigned
              Reporter:
              Andrew Chen
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: