-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Cluster Scalability
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Impact
Resharding C++ tests can hang inĀ shutdownExecutorPool() or hit teardown-time invariants when the test exits with a donor, recipient, or coordinator PrimaryOnlyService instance that has not fully completed.
Related failures include BF-43436, BF-43491, BF-43612, and BF-44484.
Root Cause
ReshardingDonorRecipientCommonTest::tearDown() calls onStepDown(), which interrupts managed instances, but it does not ensure that their asynchronous callbacks have quiesced before draining the shared executor pool. Those callbacks can continue using the mock network after the test body returns. setKillAllOperations() from SERVER-131477 does not provide a complete lifecycle barrier because some callbacks do not have an OperationContext.
The testProcessDonorFields() and testProcessRecipientFields() helpers return while their state machines remain active. The CreateDonorServiceInstance and CreateRecipientServiceInstance tests call interrupt(), but they also return without waiting for asynchronous cleanup to complete.
ReshardingCoordinatorServiceTestCommon::tearDown() has a similar ordering problem. It unblocks mock fetches and calls ConfigServerTestFixture::tearDown(), which drains the shared executor pool, before calling _registry->onShutdown(). It does not call onStepDown() unless the test body explicitly calls stepDown().
Fix
Make each fixture responsible for fully quiescing the state machines it creates.
- In ReshardingDonorRecipientCommonTest::tearDown(), interrupt and quiesce all live donor and recipient instances before draining the shared executor pool.
- In ReshardingCoordinatorServiceTestCommon::tearDown(), unblock mock fetches, call onStepDown(), wait for coordinator instances to finish, and shut down the registry before ConfigServerTestFixture::tearDown().
- Do not require individual test bodies to interrupt or run state machines to completion solely for safe fixture teardown.
Testing
- Run ProcessDonorFields* tests without per-test cleanup and verify teardown completes.
- Run CreateRecipientServiceInstance without its per-test interrupt() and verify teardown neither hangs nor triggers the recipient state-transition invariant.
- Add a coordinator test that returns while a mock fetch is blocked, without explicitly calling stepDown() or completing resharding.
- Repeat the affected tests under TSAN to exercise teardown races.
Related
- is related to
-
SERVER-127727 ReshardingDonorRecipientCommonTest can hang in shutdownExecutorPool()
-
- Closed
-
-
SERVER-128091 Fix ReshardingDonorRecipientCommonTest teardown races
-
- Closed
-
-
SERVER-131477 Kill operations during ShardingTestFixtureCommon teardown
-
- Closed
-
-
SERVER-132461 Tighten ShardingTestFixtureCommon::shutdownExecutorPool to ensure that join will not hang
-
- Closed
-