Exponential backoff and jitter in retry loops

XMLWordPrintableJSON

    • Needed
    • Hide

      Summary of necessary driver changes

      Commits for syncing spec/prose tests
      (and/or refer to an existing language POC if needed)

      Context for other referenced/linked tickets

      •  
      Show
      Summary of necessary driver changes Implement the new backpressure retry logic detailed in https://github.com/mongodb/specifications/blob/client-backpressure/source/client-backpressure/client-backpressure.md . Add opt-in token bucket support with the new adaptiveRetries URI and client option listed in https://github.com/mongodb/specifications/blob/client-backpressure/source/uri-options/uri-options.md . Token buckets are excluded from the first rollout phase of client backpressure support, see ba14b6b , copied below for convenience: WARNING: Client backpressure support will be rolled out in two separate phases. The first phase consists of backoff and jitter only, without token bucket adaptive retries. Implementing drivers MUST NOT implement token buckets in the initial release. If token bucket support has already been merged, it MUST NOT be able to be turned on by users. Commits for syncing spec/prose tests (and/or refer to an existing language POC if needed) https://github.com/mongodb/specifications/commit/1125200e4a6161d87cb5090860597eb8e8e90bf1 Context for other referenced/linked tickets  
    • $i18n.getText("admin.common.words.hide")
      Key Status/Resolution FixVersion
      CDRIVER-6092 Done 2.3.0
      CXX-3342 Backlog
      CSHARP-5723 In Code Review
      GODRIVER-3658 In Progress 2.6.0
      JAVA-5956 In Progress
      NODE-7142 Fixed 7.2.0
      PYTHON-5528 Done
      PHPLIB-1719 Backlog
      RUBY-3706 Fixed 2.24.0
      RUST-2273 Fixed 3.6.0
      $i18n.getText("admin.common.words.show")
      #scriptField, #scriptField *{ border: 1px solid black; } #scriptField{ border-collapse: collapse; } #scriptField td { text-align: center; /* Center-align text in table cells */ } #scriptField td.key { text-align: left; /* Left-align text in the Key column */ } #scriptField a { text-decoration: none; /* Remove underlines from links */ border: none; /* Remove border from links */ } /* Add green background color to cells with FixVersion */ #scriptField td.hasFixVersion { background-color: #00FF00; /* Green color code */ } #scriptField td.willNotDo { background-color: #FF0000; /* Red color code */ } /* Center-align the first row headers */ #scriptField th { text-align: center; } Key Status/Resolution FixVersion CDRIVER-6092 Done 2.3.0 CXX-3342 Backlog CSHARP-5723 In Code Review GODRIVER-3658 In Progress 2.6.0 JAVA-5956 In Progress NODE-7142 Fixed 7.2.0 PYTHON-5528 Done PHPLIB-1719 Backlog RUBY-3706 Fixed 2.24.0 RUST-2273 Fixed 3.6.0

      As part of DRIVERS-3160 Client Backpressure, we plan to make retry loops use exponential backoff and jitter to reduce the load on the server and improve goodput. Retryable reads and writes only retry once by default but can perform multiple when CSOT is enabled. The convenient transaction api (DRIVERS-1934) will also retry multiple times. These retry loops should share a common backoff and jitter policy.

      We also add an adaptive token bucket to limit load amplification during peak overload. Each time a client makes a successful request (ok:1 or a successful error like DuplicateKeyError) it deposits a fractional “token” into a bucket. Each time a request fails (ok:0 with SystemOverloaded error), the client performs retries as normal (with exponential backoff + jitter) as long as there are whole tokens available in the bucket. This approach establishes a limited memory for the operational conditions of the upstream service: if there are tokens available for retry, then the service has been healthy recently.

            Assignee:
            Jamis Buck
            Reporter:
            Shane Harvey (Inactive)
            Jib Adegunloye Jib Adegunloye
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: