The following order of actions will currently result in an unrecoverable cluster (requiring manual intervention through the wt utility):

1. Drop table
2. Crash
3. (If the storage layer has a grace period before TRIM, wait until it passes.)
4. Start up the node again
5. Retry the drop

We can implement some workarounds for this, such as WT-17996, but the root of the problem remains to be that table TRIM gets processed even though the checkpoint failed and has been abandoned. And we thus rely on the application to retry the drop to ensure that no one tries to use the table, now that the data are already gone.

We should thus implement a more robust solution to this problem, in which we effectively make removing the metadata from the shared metadata table appear to be atomic with issuing table TRIM.

*Suggested Implementation:* The high-level idea is to remove the metadata from the checkpoint's shared metadata table first, issue TRIM later, and then have a way to ensure that the TRIM can be repeated in the event of a failure. In this way, we can always safely abandon a checkpoint with an incomplete drop.

Here is the more detailed design:

• Calling WT_SESSION::drop does not issue table TRIM to the storage layer, but it otherwise works normally.
• During a checkpoint, processing a DELETE request in __wt_disagg_shared_metadata_queue_process creates a new entry in the shared metadata table with the format "drop:<table-id>". (Note that this design works well with schema epochs, as this all happens when the schema epochs intend this to happen.)
• After completing the checkpoint via the corresponding PALI call, the checkpoint can issue the TRIM. We might want to do this if this is the shutdown checkpoint or the step-down checkpoint, but otherwise it may not actually be necessary.

After the checkpoint, the table is officially dropped, as it disappears from the metadata and cannot be reopened.

At the start of the next checkpoint:

• Go through all drop: entries in the shared metadata table, issue the corresponding TRIMs, and remove those entries.
• (As a safety assertion, this code should ensure that all of those table IDs are smaller than the largest_file_id field recorded in the checkpoint.)

Note that it is allowed to TRIM the same table more than once, because table IDs are not reused by design.

In summary:

1. The first checkpoint removes the metadata, but keeps the table data around along with the drop: entries.
2. The second checkpoint completes any unfinished drops, and removes the drop: entries.

*Analysis:* Now let's see what happens in the various situations that could come up:

• Crash between the drop and the first checkpoint: The checkpoint can be safely abandoned, as if nothing happened. This is an ideal outcome.
• Crash between the first checkpoint completion and TRIM: The new primary (or the current node if it restarts in a setup without a standby node) learns about the unfinished drops and completes them.
• Crash between the first checkpoint's TRIM and the next checkpoint: The new primary learns about the (potentially) unfinished drops and completes them. This is okay, because it is perfectly legal to TRIM the same table ID more than once.
• Crash between the next checkpiont's TRIM and the completion of that checkpoint: Again, the new primary completes the drop.
• Crash after the next checkpoint completes: The table is already dropped, so we're all good here.
• Step down between the two checkpoints: Since we did not crash, the first checkpoint would have completed the TRIM. The next checkpoint by the new leader will complete the drop again, but that's okay, as TRIM can be issued more than once.

*Note that this includes a data format change.*

Upgrade: There is nothing to do. The checkpoint will look like as if all table drops are complete, which is what we will have unless we have a crashed cluster due to a bug.

Downgrade: I think the current checkpoint pick up code ignores any unknown prefixes in the metadata table. We might thus end up with a few extraneous drop: entries. Assuming that we have indeed dropped those tables with the first checkpoint, they can be safely ignored. If we did not, it means that we had an unclean shutdown, which I would really like to believe is not a standard downgrade procedure.