This will help implement the directory listing for the S3 file system. The following needs to finish before this ticket gets closed:
- AWS bucket conn class to have a function that list bucket objects:
- The function needs to take care of taking a prefix to match and listing the objects that match that prefix.
- Special handling is needed if the bucket contents are more than 1000. A test to make sure we do the right thing in such a case will also be useful. This could possibly be handled by having the function take as an argument N, the maximum number of objects to access in each iteration, defaulting N to 1000. We could test easily by keeping N low. This will also be useful when we watch to fetch just one object for implementing WT_FILE_SYSTEM.fs_directory_list_single
- Use the above function to implement S3's directory listing, i.e. the following functions:
WT_FILE_SYSTEM.fs_directory_list
WT_FILE_SYSTEM.fs_directory_list_single
WT_FILE_SYSTEM.fs_directory_list_free
Among things to consider:
- Only return the objects that match the prefix
- Take care of freeing the list
- Avoid code duplication between directory_list an directory_list_single
- Add relevant testing to the s3_store python test - following test_tiered06.py
- Add relevant statistics (No longer required - stats will be take care of by
WT-8761)