ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Execution
    • ALL
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem

      The sleepFor .onError lambda in BackfillCoordinator::markForBackfillAndScheduleIfNeeded() calls cancel() unconditionally. cancel() consumes whichever State is current, not the one the sleep was scheduled for.

      When a State's cancellation token is cancelled (for example by cancel() in the catch block of the same function), the cancelled sleep's onError runs later on an executor thread. By then a newer State is installed, so the stale callback:

      • drops the newer State's buffered queries,
      • cancels the newer State's token, and
      • if the newer State has its own scheduled sleep, that sleep's onError repeats the same step on the State after it.

      Evidence

      BF-47075: TSAN data race in BackfillCoordinatorTest.MarkForBackfillAndScheduleIfNeededDoesNotLeakErrors on enterprise-rhel8-debug-tsan-compile (v8.3 hotfix patch build, commit 8dd44bc1).

      • Write: an executor thread frees State B inside BackfillCoordinator::cancel(), called from the onError lambda (query_settings_backfill.cpp:353, :462).
      • Read: the main thread at query_settings_backfill_test.cpp:411, through the raw State* returned by the test-only peekState().

      The same unconditional cancel() is on master (checked against origin/master today) and in the v8.3 hotfix build. This sequence was derived from the TSAN stacks and the source. I have not reproduced it locally.

      Impact

      • Test: peekState() releases _mutex before the caller dereferences the pointer, so the stale cancel turns into a use-after-free in the test. peekState() has no caller outside query_settings_backfill_test.cpp.
      • Product: product code reads _state only under _mutex, so there is no use-after-free. The effect is a lost buffer and a cancelled task. Backfill is best-effort and later requests re-buffer the dropped queries.
      • Not checked: whether the coordinator outlives the this-capturing lambdas.

      Proposed fix

      • Product: capture the State's CancellationSource (or a generation id) when scheduling the sleep. In onError, cancel only if the current State is the one the sleep was scheduled for, comparing under _mutex.
      • Test: have peekState() copy the fields the tests read (buffer.size(), taskScheduled, the token) while holding _mutex, instead of returning a raw State*.
      • Add a unit test where a stale onError fires after a newer State has buffered queries, and assert the newer State is untouched.
      • Backport to every release branch that carries BackfillCoordinator.

            Assignee:
            Zixuan Zhuang
            Reporter:
            Zixuan Zhuang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: