Right now, the remote command running API accepts an ExecutorPtr as input which it uses to perform the asynchronous work. The shutdown of this executor is the only case presently when a non-RemoteCommandExecutionError can be returned by the API.

If we continue to allow for this, we should document it clearly in the code. Alternatively, we could consider using unsafeToInlineFuture to re-write executor shutdown errors into oRemoteCommandExecutionErrors, the way we do for other errors, or ban executor shutdown while RPC events are outstanding as a part of this API's contract. We could also consider  allowing our API to have it's own executor used for RPC with process-lifetime lifetime, sidestepping this issue entirely - or a mix of these ideas, such as taking in an executor as input optionally but having an 'owned' one as default.

The purpose of this ticket is to investigate the tradeoffs between these options, discuss them with stakeholders, and make a decision as to the executor semantics for this API. Then update this ticket description with the conclusion, and make any required code changes.

Update after 6WR: We're going to have the API obey the contract that it always returns ErrorCodes::RemoteCommandExecutionError; it will re-write executor shutdowns to such an error and contain the originating error information in the EEI.