[DRIVERS-1934] withTransaction API retries too frequently Created: 15/Mar/19  Updated: 25/Sep/23

Status: Backlog
Project: Drivers
Component/s: Transactions
Fix Version/s: None

Type: Spec Change Priority: Major - P3
Reporter: Shane Harvey Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: FY24Q4
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File withTransaction.py    
Issue Links:
Related
related to DRIVERS-556 Implement Convenient API for Transact... Closed
Case:

 Description   

Note that this is not really a bug because the spec was designed to retry immediately on purpose.

The withTransaction API retries immediately when encountering a TransientTransactionError. I think this may cause resource utilization problems on both the client and server in real world use cases.

As a simple example, let's say the client is running two concurrent transactions (A and B) that touch the same document. One of these transactions will error with TrasientTransactionError (caused by a WriteConflict) and immediately retry, let's assume this is transaction B. If A is still in progress then the retry of B will also fail with the same error; it will only succeed after A has completed. This can lead to hundreds of failed transactions per second.

Another example is what happens with many concurrent transactions that all contend on the same document. I've provided an example in withTransaction.py that starts 200 threads that all attempt a single with_transaction call on the same document. The output looks like this:

$ python3.7 withTransaction.py
Testing 200 threads
Finished RunOrderTransaction with 0 retry attempts
Finished RunOrderTransaction with 0 retry attempts
Finished RunOrderTransaction with 0 retry attempts
Finished RunOrderTransaction with 1 retry attempts
Finished RunOrderTransaction with 2 retry attempts
Finished RunOrderTransaction with 2 retry attempts
Finished RunOrderTransaction with 3 retry attempts
Finished RunOrderTransaction with 4 retry attempts
...
Finished RunOrderTransaction with 46 retry attempts
Finished RunOrderTransaction with 48 retry attempts
Finished RunOrderTransaction with 49 retry attempts
Finished RunOrderTransaction with 51 retry attempts
Finished RunOrderTransaction with 50 retry attempts
Finished RunOrderTransaction with 51 retry attempts
...
Finished RunOrderTransaction with 108 retry attempts
Finished RunOrderTransaction with 112 retry attempts
Finished RunOrderTransaction with 116 retry attempts
Finished RunOrderTransaction with 112 retry attempts
Finished RunOrderTransaction with 114 retry attempts
Finished RunOrderTransaction with 116 retry attempts
Finished RunOrderTransaction with 118 retry attempts
Finished RunOrderTransaction with 116 retry attempts
All threads completed after 21.644919872283936 seconds

One solution to this problem could be adding a delay before attempting to retry. When I change with_transaction to have a 250 millisecond retry delay the withTransaction script completes much faster and with much fewer retry attempts:

$ python3.7 withTransaction.py
Testing 200 threads
Finished RunOrderTransaction with 0 retry attempts
...
Finished RunOrderTransaction with 27 retry attempts
Finished RunOrderTransaction with 28 retry attempts
Finished RunOrderTransaction with 30 retry attempts
Finished RunOrderTransaction with 30 retry attempts
Finished RunOrderTransaction with 31 retry attempts
Finished RunOrderTransaction with 33 retry attempts
All threads completed after 10.54933214187622 seconds

Note that a fixed retry delay is only one solution. We can also investigate others, like exponential backoff or something else.



 Comments   
Comment by Scott L'Hommedieu (Inactive) [ 29/Apr/19 ]

None yet.  Beta is just getting started.

Comment by Ian Whalen (Inactive) [ 29/Apr/19 ]

scott.lhommedieu any feedback so far on this?

Comment by Jeremy Mikola [ 01/Apr/19 ]

scott.lhommedieu: Any follow-up on this from beta users?

Comment by Jeremy Mikola [ 15/Mar/19 ]

scott.lhommedieu: Is this worth discussing with users during your outreach or the beta program? That might provide some guidance on what type of solution would be best (e.g. fixed delay, back-off).

Comment by Shane Harvey [ 15/Mar/19 ]

Also bear in mind that this python example may not be the best because it uses threads which all contend on the Python GIL (global interpreter lock). The same example written in Java/Go/C etc.. may show different results.

Generated at Thu Feb 08 08:24:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.