Create S3 sink jstest

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 8.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Atlas Streams
    • Fully Compatible
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      This ticket involves setting up jstests to run pipelines with the S3 emit operator against a locally running S3-compatible container.

       

      We can use this docker image

      https://hub.docker.com/r/minio/minio

       

      (though i have noticed some weirdness when calling PutObject using the AWS SDK against it)

       

      I tried using this test in aws_test.cpp and adapting it to suit my needs for testing.

      TEST(AWS, AWSS3Test) {
          SDKOptions options;
          options.loggingOptions.logLevel = Utils::Logging::LogLevel::Error;
          InitAPI(options);    
          Aws::Client::ClientConfiguration clientConfig;
          
          // Optional: Set to the AWS Region (overrides config file).
          clientConfig.region = "us-east-2";    
          
          auto creds = Auth::AWSCredentials();
          // set this up according to the directions on the minio guide         
          creds.SetAWSAccessKeyId("");
          creds.SetAWSSecretKey("");    
          creds.SetSessionToken("");    
      
          Aws::S3::S3Client client(emptyCreds, nullptr, clientConfig);    
          std::string fileName = "second";
          std::string bucketName = "andrewchentest";
          Aws::S3::Model::PutObjectRequest request;
          request.SetBucket(bucketName);
          request.SetKey(fileName);    
          
          const std::shared_ptr<Aws::IOStream> inputData = Aws::MakeShared<Aws::StringStream>("");
          *inputData << "time to cook";    
          request.SetBody(inputData);    
      
          Aws::S3::Model::PutObjectOutcome outcome = client.PutObject(request);      if (!outcome.IsSuccess()) {
              std::cerr << "Error: putObject: " << outcome.GetError().GetMessage() << std::endl;
          } else {
              std::cout << "Added object '" << fileName << "' to bucket '" << bucketName << "'.";
          }
          ShutdownAPI(options);
      }
       

              Assignee:
              Andrew Chen
              Reporter:
              Andrew Chen
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: