As we start to ramp up testing of tiered storage, we will have potentially unwanted sharing of objects.  Some statement of the problem is below, along with two suggestions for detecting trouble: First having a GUID created for each new connection that is stored both in the local connection home directory and the cloud bucket (using our prefix).  Second, require storage_source to implement a "no-overwrite" policy for objects stored in the cloud (or simulated cloud).

The problem:

Consider s3 test runs, where evergreen environment variables specify access keys and bucket names to use.  Since evergreen controls these, it can't get mixed up (can it?).  But what if we have a program like timestamp_abort, or test/format that is doing lots of individual runs in a loop.  These may be in the same evergreen job.  Currently, test programs know to remove/recreate the TESTDIR or RUNDIR at the beginning of each run and away we go.  Obviously cloud objects left behind from a previous run need to be cleaned up as well.

dir_store does not usually have that particular problem, since the "bucket" directory is generally relative to the WT home directory, and so can be removed at the same time.  But dir_store can use absolute paths, which we may want to do at some point to test flushing and accepting nodes in tandem.  So the issues will be very similar to cloud storage.

So one step obviously will need to be applied in either scripts that drive the multiple tests or the tests themselves, that know to remove objects from a bucket either before each run or after any successful run (preserving any needed artifacts on failures).  We can/should do this as we're writing tests or adapting them to be used for tiered storage.

Detection:

Another step we might consider is some detection of trouble at different levels.  First, at the connection level.  WiredTiger has protections against multiple simultaneous connections on a single directory.  A connection must be fully closed before another connection is allowed, and then that connection starts up with the metadata from the first run - it cannot start "fresh" with empty metadata.  And yet, even when we are starting a newly created WT home directory, we can attach to a cloud directory that has other stuff in it (from previous runs or simultaneous runs).  "prefixes" for names in buckets can help with that, but we don't check for instance that there are no objects with our prefix when we are first starting up.  Maybe with each fresh WT connection (i.e. empty home directory), we create a GUID that we put into the metadata, and create a bucket/prefix_WiredTigerGUID.txt object.  If it exists, we're about to trample on another run.  On subsequent reopens, we make sure the GUID matches.

At the file/object level.  When we push say pfx-T00002-0000000004.wtobj to the cloud, we don't necessarily check that that file already exists.  (I don't think dir_store does this, I don't know the AWS APIs well enough to know if s3_store is doing this).  I think refusing to overwrite should be the default.  That will also help us stay out of trouble.

At the block level, I think we already have good protections - if we ask to read an object and we somehow have someone else's object, we should get a checksum error.