-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
None
-
Replication
-
Fully Compatible
-
ALL
-
Repl 2026-07-20, Repl 2026-08-03
-
200
-
None
-
None
-
None
-
None
-
None
-
None
-
None
BF-44190: _runReadyNetworkOperations_inlock drops canceled alarms after executor yield
The race
TransactionCoordinator completion follows this chain after the last commit response is delivered:
- deleteCoordinatorDoc() completes → outer .getAsync() lambda fires on the executor thread
- Lambda calls _scheduler->shutdown() → iterates pending alarms and calls .cancel() on each
- cancel() inserts the alarm ID into NetworkInterfaceMock::_canceledAlarms
- _canceledAlarms must be drained by the network thread: each entry calls alarm.cancel(), which resolves deadlineFuture, which unblocks _done(), which sets _completionPromise
The problem is in _runReadyNetworkOperations_inlock ([network_interface_mock.cpp:765–775|vscode-webview://0a4n1vfqoj7somk3nd5dip8dksek0l7uaah3evkkq8spmbsgsi6m/src/mongo/executor/network_interface_mock.cpp]):
{{Deliver response → set kExecutorThread → yield to executor → RETURN
↑ executor populates _canceledAlarms here
but nobody drains them}}
After yielding to the executor (line 769–774), the function returned unconditionally. The executor ran step 2–3 above during that yield, but _canceledAlarms was never checked again. Result: deadlineFuture never resolved, _completionPromise was never set, and anything awaiting onCompletion() hung forever.
Log evidence (from CI failures, e.g. 537bc1963e9d27bced15afbea7447660.txt):
{{"Deleted coordinator doc" ← log ID 22474, step 1 completed
← log ID 22447 "Two-phase commit completed" ABSENT}}
The coordinator doc was deleted (chain reached the lambda) but _done() never fired.
This is a race because in production the executor runs on a real thread — if it finishes populating _canceledAlarms before the test pumps again, the second pump drains them fine. Under resource pressure or timing variance, the pump runs first (finding _canceledAlarms empty), then the executor populates it, and nobody pumps again.
The 10-day quiet period (July 3–13) correlated with the revert of SERVER-128711, which had widened the race window by adding extra work in the completion chain.
- is related to
-
SERVER-128711 Migrate TransactionCoordinatorDocument
-
- Closed
-