Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-8712

Add a method to list bucket contents and implement directory_list

    • Type: Icon: New Feature New Feature
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT11.0.0, 5.3.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • 5
    • Storage - Ra 2022-02-07

      This will help implement the directory listing for the S3 file system. The following needs to finish before this ticket gets closed:

      • AWS bucket conn class to have a function that list bucket objects:
      • The function needs to take care of taking a prefix to match and listing the objects that match that prefix.
      • Special handling is needed if the bucket contents are more than 1000. A test to make sure we do the right thing in such a case will also be useful. This could possibly be handled by having the function take as an argument N, the maximum number of objects to access in each iteration, defaulting N to 1000. We could test easily by keeping N low. This will also be useful when we watch to fetch just one object for implementing WT_FILE_SYSTEM.fs_directory_list_single
      • Use the above function to implement S3's directory listing, i.e. the following functions:
        WT_FILE_SYSTEM.fs_directory_list
        WT_FILE_SYSTEM.fs_directory_list_single
        WT_FILE_SYSTEM.fs_directory_list_free

      Among things to consider:

      • Only return the objects that match the prefix
      • Take care of freeing the list
      • Avoid code duplication between directory_list an directory_list_single
      • Add relevant testing to the s3_store python test - following test_tiered06.py
      • Add relevant statistics (No longer required - stats will be take care of by WT-8761)

            Assignee:
            fiona.ochee@mongodb.com Fiona O'Chee (Inactive)
            Reporter:
            sulabh.mahajan@mongodb.com Sulabh Mahajan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: