RateLimiter param updates forgive debt; interrupt returns use new rate

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Workload Resilience
    • ALL
    • WR Prioritized list
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      BasicTokenBucket::reset recomputes zeroTime from available(now), which clamps negative balances to 0. So a setParameter on the ingress or egress limiter (both use the non-preserving variant; only the write throttler uses updateRateParametersPreservingBalance) instantly wipes all queued debt. Sleeping waiters aren't woken early (their naps are precomputed), but new acquires see an empty bucket and are admitted immediately, decoupled from the old waiters' schedule. Additionally, returnTokens pulls zeroTime back by numTokens / rate_ using the current rate. That means if the rate changed between borrow and interrupt-return, the undo doesn't cancel the borrow, after a rate decrease it over-returns, and after an increase it under-returns.

      Curiously, this behavior might be helping us today in our SLO journey. The impact of this bug is that the IRRL queue behaves a little more like a LIFO queue (new arrivers can "barge" and get a token instead of being queued, if they are lucky). Fixing this bug makes us more truly FIFO, and we will see increased latency as a consequence.

      Suggested solution: use the preserving-balance variant everywhere (old waiters still aren't retimed, but new borrows at least queue behind outstanding debt). Additionally, consider storing the borrow-time rate in the DeferredToken so interrupt returns undo the exact borrow.

            Assignee:
            Unassigned
            Reporter:
            Matt Broadstone
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: