Thread pool task executor when it handles exhaust commands can leak resources. We noticed the leakage with SingleServerDiscoveryMonitor which is the only consumer at this moment which uses thread pool task executor to schedule exhaust commands . Below is the racy sequence with exhaust command workflow.
1. Thread 1: Starts processing the streamable hello cmd response and runs line 842, and returns true (i.e, cbState is neither canceled nor finished at this point); and the thread pauses.
2. Thread 2: Cancels the exhaust command which will cancel the cbState and resets the callback in cbstate to release any resources the callback is holding.
3. Thread 1: Thread resumes, then acquires the executor mutex lock and sets the callback in cbState back to the original callback function. And, this can cause memory leak.
In the BF, we were ending up with indirect leak due to this circular reference: SingleServerDiscoveryMonitor holds a reference to TaskExecutor::CallbackHandle and which in-turn holds a reference to ThreadPoolTaskExecutor::CallbackState and that contains CallbackFn which holds a reference to SingleServerDiscoveryMonitor.