ExportXMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Cluster Scalability
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Impact

      Resharding C++ tests can hang inĀ shutdownExecutorPool() or hit teardown-time invariants when the test exits with a donor, recipient, or coordinator PrimaryOnlyService instance that has not fully completed.

      Related failures include BF-43436, BF-43491, BF-43612, and BF-44484.

      Root Cause

      ReshardingDonorRecipientCommonTest::tearDown() calls onStepDown(), which interrupts managed instances, but it does not ensure that their asynchronous callbacks have quiesced before draining the shared executor pool. Those callbacks can continue using the mock network after the test body returns. setKillAllOperations() from SERVER-131477 does not provide a complete lifecycle barrier because some callbacks do not have an OperationContext.

      The testProcessDonorFields() and testProcessRecipientFields() helpers return while their state machines remain active. The CreateDonorServiceInstance and CreateRecipientServiceInstance tests call interrupt(), but they also return without waiting for asynchronous cleanup to complete.

      ReshardingCoordinatorServiceTestCommon::tearDown() has a similar ordering problem. It unblocks mock fetches and calls ConfigServerTestFixture::tearDown(), which drains the shared executor pool, before calling _registry->onShutdown(). It does not call onStepDown() unless the test body explicitly calls stepDown().

      Fix

      Make each fixture responsible for fully quiescing the state machines it creates.

      • In ReshardingDonorRecipientCommonTest::tearDown(), interrupt and quiesce all live donor and recipient instances before draining the shared executor pool.
      • In ReshardingCoordinatorServiceTestCommon::tearDown(), unblock mock fetches, call onStepDown(), wait for coordinator instances to finish, and shut down the registry before ConfigServerTestFixture::tearDown().
      • Do not require individual test bodies to interrupt or run state machines to completion solely for safe fixture teardown.

      Testing

      • Run ProcessDonorFields* tests without per-test cleanup and verify teardown completes.
      • Run CreateRecipientServiceInstance without its per-test interrupt() and verify teardown neither hangs nor triggers the recipient state-transition invariant.
      • Add a coordinator test that returns while a mock fetch is blocked, without explicitly calling stepDown() or completing resharding.
      • Repeat the affected tests under TSAN to exercise teardown races.

      Related

            Assignee:
            Unassigned
            Reporter:
            Abdul Qadeer
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: