Range deleter can hang node stepUp/shutdown when an in-flight deletion batch is not interrupted on stepdown

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • None
    • Cluster Scalability
    • Fully Compatible
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      ReadyRangeDeletionsProcessor runs each deletion batch on a dedicated alternate client ("range-deleter-batch") whose OperationContext is a CancelableOperationContext created from HierarchicalCancelableOperationContextFactory:

      That batch opCtx is killed on cancellation via an executor-scheduled continuation.

      On stepdown, RangeDeleterService::_stopService shuts the executor down before calling ReadyRangeDeletionsProcessor::shutdown(). When the parent cancellation token subsequently fires, the .thenRunOn(_executor) continuation runs with a cancelled status, so the markKilled body is skipped and the batch opCtx is never interrupted.

      If a deletion batch is blocked mid-flight due to some reason when the stepdown lands the range-deleter thread never returns. Consequences cascade:

       

            Assignee:
            Unassigned
            Reporter:
            Abdul Qadeer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: