Summary

For ASC tests, we can view the core dumps/data files to inspect/debug a test failure.
For DSC tests, there is currently no equivalent because the data lives on the SLS nodes.

Background / Context

• All the data is in the volumes used by the SLS containers. if we can add a little thing to the evergreen setup to have it tarball those up, we'd be able to download + inspect it
  • the one unfortunate gotcha: localstack doesn't keep its data on disk, its all in-memory (having persistent data is an enterprise feature and we currently don't pay for that). so whatever data is in object storage won't be available in the tarball. that may or may not matter depending on the case you're debugging.
  • It's pretty unlikely that any of our tests need to actually read from object storage anyway and so the data in page servers should have everything to see what was going on in the cluster, but worth noting.

• The  src/mongo/db/modules/atlas/jstests/disagg_storage/libs/recreate.py script (context available in this doc and https://jira.mongodb.org/browse/SERVER-116124) should address the problem for tests within the disagg_storage suite only by replaying the each op/phylog observed throughout the test execution). We could also try to generalize this script for more suites.
• The WT team shared some instructions to build & use an util to inspect data at their level - but the process is more cumbersome it seeems to be  only applicable to "live clusters"

Proposed Solution

Requirements

• create a way for server engineers to easily inspect the data state of a test failure

Impact

Impacts every team that works on disagg on server, which includes all of RSS, CAR, CS, Query, etc