Diagnose potential rate limiter failure mode with expiring waiters leading to admission storms

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Fix Not Needed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Workload Resilience
    • ALL
    • Workload Resilience 2026-08-31
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Our connection_storm workloads started failing after increasing our static queue length for CRL. We think there might be a bug in the token bucket implementation that leads to potentially faulty token accounting for jobs on the queue that expire (either maxTimeMS for IRRL, or client timeout for CRL) that may lead to admission storms.

            Assignee:
            Jiaqi Tian
            Reporter:
            Jason Chan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: