Major - P3 Background:
Quote from Max Hirschhorn on Slack:
I noticed while debugging a unit test failure for one of the resharding primary-only services that Instance::run() is scheduled as a task within the _scopedExecutor. This means it is possible for the following sequence to occur:
- getOrCreate() is called and caller now holds shared_ptr<Instance>
- _insertNewInstance() schedules task to call Instance::run() but it doesn't run yet due to thread scheduling
- stepdown occurs
- _scopedExecutor is shut down and won't run any tasks
- Instance::run() is never called
Symptoms:
The completion promise is never fulfilled because Instance::run() is never called. This trips the invariant that the completion promise is fulfilled.
Solution:
Remove the invariant. It's okay for the completion promise to not be fulfilled if the _scopedExecutor has not called Instance::run().