By far the biggest time consumer during the gathering of handles is the time spent to parse the checkpoint strings from the metadata into the ckpt array via wt_meta_ckptlist_get. By far the most common scenario is that the checkpoint thread loads the most recent checkpoint WiredTigerCheckpointN into ckpt[0] and then allocates one for the in-progress checkpoint WiredTigerCheckpointN+1 at ckpt[1]. Checkpoint then fills in and generates the metadata string for ckpt[1]. Then on the next checkpoint it reads in the string and parses WiredTigerCheckpointN+1 into ckpt[0] and allocates one at ckpt[1] for WiredTigerCheckpointN+2. And so on. If there are no metadata changes for this dhandle other than this pattern, we could store the most recent ckpt structures in the dhandle between checkpoints so that on the next checkpoint we don’t have to reread and reparse the strings.

If anything changes the metadata at all then those stored ckpt structures are invalidated and we would continue with the slow path we have today. But if the pattern holds, then we may have a very rapid gathering of handles. Code paths that would invalidate these stored structures include (but are not limited to) the WT_SESSION::alter API, using named checkpoints in any way, any code path that checkpoints a file individually.

The also-common path of taking checkpoints while a backup cursor is opened should also be part of this work. In that case we would simply have an extra checkpoint stored: the backup checkpoint that we retain until the cursor is closed. This change should work with backups.

I believe, but have not fully thought through, that we only need to keep the most recent checkpoint (i.e. the one being generated this time) for comparison or basis for the next checkpoint. Maybe we need to keep them all stored (at most 2 old ones plus the current one we’re creating) so that we can do the deletion checks, etc.

However avoiding the string parsing will be the biggest win when you have thousands of files to walk through.