Fix NetworkInterfaceMock cancellation race that can hang transaction coordinator tests

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.1.0-rc0, 9.0.1, 9.0.2
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • Fully Compatible
    • v9.0, v8.3
    • CAR Team 2026-08-03, CAR Team 2026-08-17
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Unit tests that use `NetworkInterfaceMock` can hang indefinitely when an alarm is canceled.
      In CI this surfaced as `db_s_shard_server_part3_test` timing out and being killed
      (BF-45092 / BFG-3474784), with the stack stuck in `SharedStateBase::wait` under
      `TransactionCoordinator::onCompletion()`.

      The cause was a missed wakeup in `NetworkInterfaceMock::setAlarm()`. The `onCancel` callback
      did not resolve the alarm's promise; it only recorded the alarm id in `_canceledAlarms`, leaving
      resolution to the next pass of `_runReadyNetworkOperations_inlock()`. Because the mock only makes
      progress while a test actively drives the network thread, any thread that cancels an alarm and then
      blocks waiting on it is never woken. The `TransactionCoordinator` cleanup path does exactly this,
      so the test blocked forever on `onCompletion()` after all network responses had been delivered.

      SERVER-131498 previously addressed a narrower form of this by looping in
      `_runReadyNetworkOperations_inlock()` to drain `_canceledAlarms` after yielding to the executor.
      That reduced the window but kept the dependency on the pump, and the hang still reproduced on
      master afterwards (locally, at roughly iteration 1400 of
      `TransactionCoordinatorTest.RunCommitProducesCommitDecisionOnTwoCommitResponses`).

      timeout 660s bazel run +db_s_shard_server_part3_test --   --gtest_filter=TransactionCoordinatorTest.RunCommitProducesCommitDecisionOnTwoCommitResponses   --gtest_repeat=10000   --gtest_break_on_failure
      

            Assignee:
            Igor Praznik
            Reporter:
            Igor Praznik
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: