Investigate (additional) client-side throttling strategies for mitigating retry-storms

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Critical - P2
    • None
    • Component/s: Backpressure

      Summary

      As part of the retryAfterMS (baseBackoffMS) work, the team identified a follow-up problem: applications may still create retry storms by immediately retrying surfaced overload errors in tight loops, even after the initial retryAfterMS work ships.

      DRIVERS should investigate whether client-side throttling could produce availability improvements. Report findings and then circulate this with folks in the larger IWM effort. 

      Motivation

      Several possible client-side approaches have already come up, including tar-pitting (waitForMS), queueing, smarter server selection, or other throttling behaviors, but none of them have yet been validated for efficacy or suitability on the 9.0 timeline.

      At the same time, there is concern from the server side that naive user retry loops are a realistic app pattern and could continue to degrade overloaded systems if the client surfaces errors immediately without any additional mitigation.

      This ticket is therefore to investigate the space, define candidate approaches, and determine whether any client-side throttling mechanism is worth pursuing after retryAfterMS ships.

      Who is the affected end user?

      Clients leveraging an application with a MongoDB Driver library 

      How does this affect the end user?

      Lowered availability can mean a subpar user experience for the app they're working with

      How likely is it that this problem or use case will occur?

      That is part of what we're exploring

      If the problem does occur, what are the consequences and how severe are they?

      Unavailability

      Is this issue urgent?

      This is a fast-follow item after the release of IWM 9.0

      Is this ticket required by a downstream team?

      Upstream* Availability & Server team

      Is this ticket only for tests?

      Just an investigation ticket.

      Acceptance Criteria

      • Determine whether application-driven manual retry loops are a meaningful enough source of overload amplification to justify client-side mitigation work.
      • Evaluate candidate client-side throttling strategies and their trade-offs.
      • Produce a recommendation on whether to proceed with a concrete DRIVERS design, defer, or solve this via documentation/comms instead.

            Assignee:
            Unassigned
            Reporter:
            Jib Adegunloye
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: