Uploaded image for project: 'Drivers'
  1. Drivers
  2. DRIVERS-1934

withTransaction API retries too frequently

    • Type: Icon: Spec Change Spec Change
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Component/s: Transactions

      Note that this is not really a bug because the spec was designed to retry immediately on purpose.

      The withTransaction API retries immediately when encountering a TransientTransactionError. I think this may cause resource utilization problems on both the client and server in real world use cases.

      As a simple example, let's say the client is running two concurrent transactions (A and B) that touch the same document. One of these transactions will error with TrasientTransactionError (caused by a WriteConflict) and immediately retry, let's assume this is transaction B. If A is still in progress then the retry of B will also fail with the same error; it will only succeed after A has completed. This can lead to hundreds of failed transactions per second.

      Another example is what happens with many concurrent transactions that all contend on the same document. I've provided an example in withTransaction.py that starts 200 threads that all attempt a single with_transaction call on the same document. The output looks like this:

      $ python3.7 withTransaction.py
      Testing 200 threads
      Finished RunOrderTransaction with 0 retry attempts
      Finished RunOrderTransaction with 0 retry attempts
      Finished RunOrderTransaction with 0 retry attempts
      Finished RunOrderTransaction with 1 retry attempts
      Finished RunOrderTransaction with 2 retry attempts
      Finished RunOrderTransaction with 2 retry attempts
      Finished RunOrderTransaction with 3 retry attempts
      Finished RunOrderTransaction with 4 retry attempts
      ...
      Finished RunOrderTransaction with 46 retry attempts
      Finished RunOrderTransaction with 48 retry attempts
      Finished RunOrderTransaction with 49 retry attempts
      Finished RunOrderTransaction with 51 retry attempts
      Finished RunOrderTransaction with 50 retry attempts
      Finished RunOrderTransaction with 51 retry attempts
      ...
      Finished RunOrderTransaction with 108 retry attempts
      Finished RunOrderTransaction with 112 retry attempts
      Finished RunOrderTransaction with 116 retry attempts
      Finished RunOrderTransaction with 112 retry attempts
      Finished RunOrderTransaction with 114 retry attempts
      Finished RunOrderTransaction with 116 retry attempts
      Finished RunOrderTransaction with 118 retry attempts
      Finished RunOrderTransaction with 116 retry attempts
      All threads completed after 21.644919872283936 seconds
      

      One solution to this problem could be adding a delay before attempting to retry. When I change with_transaction to have a 250 millisecond retry delay the withTransaction script completes much faster and with much fewer retry attempts:

      $ python3.7 withTransaction.py
      Testing 200 threads
      Finished RunOrderTransaction with 0 retry attempts
      ...
      Finished RunOrderTransaction with 27 retry attempts
      Finished RunOrderTransaction with 28 retry attempts
      Finished RunOrderTransaction with 30 retry attempts
      Finished RunOrderTransaction with 30 retry attempts
      Finished RunOrderTransaction with 31 retry attempts
      Finished RunOrderTransaction with 33 retry attempts
      All threads completed after 10.54933214187622 seconds
      

      Note that a fixed retry delay is only one solution. We can also investigate others, like exponential backoff or something else.

            Assignee:
            Unassigned Unassigned
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: