-
Type:
Task
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Cluster Scalability
-
Fully Compatible
-
ClusterScalability 14Sep-28Sep, ClusterScalability 28Sep-12Oct
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Impact
Resharding C++ tests can hang in shutdownExecutorPool() or hit teardown-time invariants when the test exits with a donor, recipient, or coordinator PrimaryOnlyService instance that has not fully completed.
Related failures include BF-43436, BF-43491, BF-43612, and BF-44484.
Root Cause
ReshardingCoordinatorServiceTestCommon::tearDown() never steps down the coordinator instances, and it calls _registry->onShutdown() only after ConfigServerTestFixture::tearDown(). That base teardown shuts down the executor pool, ShardRegistry and CatalogCache, and clears ShardingCatalogManager and Grid, so coordinators can keep running against torn-down sharding state. ReshardingDonorRecipientCommonTest already uses the correct order: step down, shut down executors, join instances, then tear down the fixture. Its per-test interrupt() calls are redundant.
Fix
Make each fixture responsible for fully quiescing the state machines it creates.
- In the coordinator fixture: step down, shut down the executor pool, then call onShutdown() before ConfigServerTestFixture::tearDown(). Remove the redundant per-test interrupt() calls.
- Do not require individual test bodies to interrupt or run state machines to completion solely for safe fixture teardown.
Testing
- Run ProcessDonorFields* tests without per-test cleanup and verify teardown completes.
- Run CreateRecipientServiceInstance without its per-test interrupt() and verify teardown neither hangs nor triggers the recipient state-transition invariant.
- Add a coordinator test that returns while a mock fetch is blocked, without explicitly calling stepDown() or completing resharding.
- Repeat the affected tests under TSAN to exercise teardown races.
Related
- is related to
-
SERVER-127727 ReshardingDonorRecipientCommonTest can hang in shutdownExecutorPool()
-
- Closed
-
-
SERVER-128091 Fix ReshardingDonorRecipientCommonTest teardown races
-
- Closed
-
-
SERVER-131477 Kill operations during ShardingTestFixtureCommon teardown
-
- Closed
-
-
SERVER-132461 Tighten ShardingTestFixtureCommon::shutdownExecutorPool to ensure that join will not hang
-
- Closed
-