Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8927

S3 Download objects from the bucket on test failure or preserve

    • 8
    • Storage - Ra 2022-05-16, Storage - Ra 2022-05-30

      Design a scheme to help diagnose failures in S3 extension testing.

      There is some work required to make sure that failures in the S3 extension are diagnosable. Here are a few things to consider:

      • Python and the unit testing for S3 generate unique prefix for the bucket(s) used. This is done to make sure that the tests being run in parallel by the same or a different user do not collide. These prefixes are prepended to the object files the test creates. I can see a need to access these files if a test fails. There will be a need to identify the objects generated by a test, and a means to access them. Ideally we could generate the prefixes in such a manner, the objects generated by a test are grouped together. S3 offers the use of '/' separator to simulate the filesystem hierarchy. We could utilise that method to separate the test runs in various 'folders' and 'subfolders'
      • S3 testing doesn't clear the artifacts that it generates. At the moment, we use a lifecycle rule on the buckets to expire the objects. Alternative to that would be to use a test helper to delete the objects we do not need.
      • We will like to make it easy for a user to locate the test object if a test fails. The prefix used (or the 'folder' path in the hierarchical prefixing), a means to download the objects, or to archive them separately will be useful. Ideally we could write routines such that evergreen can download the failing test artifacts and upload them along with the regular test files in a single place.
      • Figure out how to load the debug symbols for the extension when using a debugger. Write a wiki that could help developers debug the extension if needed.
      • Is this issue urgent?
      • Documentation update
        This ticket will likely produce a wiki or other means to detail the scheme we come up with.

            sean.watt@mongodb.com Sean Watt
            sulabh.mahajan@mongodb.com Sulabh Mahajan
            0 Vote for this issue
            3 Start watching this issue