-
Type:
Bug
-
Resolution: Works as Designed
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Execution
-
ALL
-
0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Problem
The sleepFor .onError lambda in BackfillCoordinator::markForBackfillAndScheduleIfNeeded() calls cancel() unconditionally. cancel() consumes whichever State is current, not the one the sleep was scheduled for.
When a State's cancellation token is cancelled (for example by cancel() in the catch block of the same function), the cancelled sleep's onError runs later on an executor thread. By then a newer State is installed, so the stale callback:
- drops the newer State's buffered queries,
- cancels the newer State's token, and
- if the newer State has its own scheduled sleep, that sleep's onError repeats the same step on the State after it.
Evidence
BF-47075: TSAN data race in BackfillCoordinatorTest.MarkForBackfillAndScheduleIfNeededDoesNotLeakErrors on enterprise-rhel8-debug-tsan-compile (v8.3 hotfix patch build, commit 8dd44bc1).
- Write: an executor thread frees State B inside BackfillCoordinator::cancel(), called from the onError lambda (query_settings_backfill.cpp:353, :462).
- Read: the main thread at query_settings_backfill_test.cpp:411, through the raw State* returned by the test-only peekState().
The same unconditional cancel() is on master (checked against origin/master today) and in the v8.3 hotfix build. This sequence was derived from the TSAN stacks and the source. I have not reproduced it locally.
Impact
- Test: peekState() releases _mutex before the caller dereferences the pointer, so the stale cancel turns into a use-after-free in the test. peekState() has no caller outside query_settings_backfill_test.cpp.
- Product: product code reads _state only under _mutex, so there is no use-after-free. The effect is a lost buffer and a cancelled task. Backfill is best-effort and later requests re-buffer the dropped queries.
- Not checked: whether the coordinator outlives the this-capturing lambdas.
Proposed fix
- Product: capture the State's CancellationSource (or a generation id) when scheduling the sleep. In onError, cancel only if the current State is the one the sleep was scheduled for, comparing under _mutex.
- Test: have peekState() copy the fields the tests read (buffer.size(), taskScheduled, the token) while holding _mutex, instead of returning a raw State*.
- Add a unit test where a stale onError fires after a newer State has buffered queries, and assert the newer State is untouched.
- Backport to every release branch that carries BackfillCoordinator.