It is incorrect for CancelRemotelyTimeout to wait for the _killOperations command issued by cancelCommand to be killed when it times out since we don't kill operations without operation keys (i.e. without hedged options). The test rarely fails or hangs since the currentOp check and the _killOperations command both have the timeout of 1 second. However, there is a race between when the _killOperations operation starts and when the currentOp check starts. That is, there is a chance that the currentOp check returns without waiting because there is no _killOperations operation running yet, and so the test fails as the number of timed out commands is zero. To solve this, we should remove the currentOp check and instead use a failpoint to wait for the _killOperations command to timeout. That failpoint should be placed at the end of NetworkInterfaceTL::tryFinish and have a predicate that only returns true when command name and error code match the response status and command name for the request.
The failpoint "networkInterfaceAfterAcquireConn" was added to ensure that cancelCommand does not start running until the command acquires a connection (otherwise, no _killOperations command will be issued). However, it does not check the command name so other commands in the background can also enter this failpoint if they happen to run while the test is in this block. So to avoid this, we should replace the failpoint with a function similar to waitForCommand that runs currentOp repeatedly until there is a matching operation is running.
Bug
Major - P3