Summary:

Fix problems in prepared transaction API to fail attempted operations after a prepare without forcing the transaction to rollback, and panic the system if a prepared transaction cannot be subsequently committed.

The changes in WT-5324 have a problem: the test for WT_SESSION.rollback_transaction as the caller in the __wt_txn_err_chk() function call was wrong, always returning success.

Using the correct test reveals an underlying problem, we need to allow at least WT_CURSOR.reset and WT_CURSOR.close calls in cases where an error has already occurred within a transaction.

Additionally, under some conditions, prepared transactions can be rolled back, which violates transactional semantics.