Background

Found in code review of WT-18922, whose blind_remove ENOTSUP check in __cursor_reuse_or_init makes problem 1 reachable from user input, so WT-18922 should not merge first. Related: WT-18941, WT-18942.

Problems

When a cursor fails partway through being opened, cleaning it up can crash the process:

1. A cursor that never finished opening is cached. _curfile_create and wt_clayered_open set WT_CURSTD_CACHEABLE before wt_cursor_init. If init fails, the error path clears the dhandle and closes the cursor, and wti_cursor_cache_release tries to cache it. _wti_cursor_cache then fails its dhandle != NULL assert; release builds segfault there.
2. Closing a layered cursor that failed to open dereferences a NULL dhandle. _wt_clayered_open's error path clears clayered->dhandle before calling clayered_close, whose API entry sets session->dhandle from it. The WT_ASSERT_ALWAYS in _clayered_close_int then reads session->dhandle->type through NULL, whether or not the cursor is cacheable.
3. A failed layered cursor allocation dereferences a NULL cursor. The same error path writes clayered->dhandle before checking clayered != NULL.

Reachability

• Develop: only through a failed memory allocation, either of a layered cursor, or of the dump cursor for a layered cursor opened with dump (it stays cacheable until that allocation succeeds).
• WT-18922: problem 1 is reachable by opening a non-layered table with blind_remove=true when the session has no cached cursor for it. Problems 2 and 3 are not, because the check always passes for layered cursors.

Reproduction

On the WT-18922 branch, session.open_cursor('table:plain', None, 'blind_remove=true') on a newly created plain table aborts in _wti_cursor_cache, called from curfile_create's error path. Problem 2 reproduces under ASan by injecting a failure into layered cursor init: both cacheable and non-cacheable opens segfault in _clayered_close_int. Problem 3 is from reading the code.

Fix

1. __wti_cursor_cache_release: don't cache a cursor without WT_CURSTD_OPEN. Cached and reopened cursors always have it, so cache reuse is unchanged.
2. _clayered_close_int: allow a NULL session->dhandle, as _curfile_close does.
3. __wt_clayered_open: in the error path, clear clayered->dhandle only after checking clayered is not NULL.

Verification

With the fixes, the problem 1 reproduction returns ENOTSUP and the injected layered failures return their error, all clean under ASan with leak detection. On develop, 1,470 cursor, cursor-cache, dump and sweep tests pass with fix 1, and 922 layered cursor and schema tests pass with all three.

Testing

Once this is in, WT-18922 can add a regression test to test_layered_cursor31: a blind_remove=true open on the plain table from a new session returns ENOTSUP, and the next open does not reuse a cached cursor. A test on develop alone needs a forced failure, for example a failpoint or a Catch2 test.