[SERVER-39704] Allow mongos to retry on stale version and snapshot errors within a transaction Created: 20/Feb/19  Updated: 17/Nov/23

Status: Needs Scheduling
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Jack Mulrow Assignee: Backlog - Cluster Scalability
Resolution: Unresolved Votes: 4
Labels: ShardedTxn:FutureOptimizations, pm-564, sharding-causes-bfs-hard
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-37207 Only retry failed writes in a batch o... Backlog
is depended on by RUST-900 Remove extraneous calls to distinct i... Blocked
is depended on by CDRIVER-3301 Unskip txn tests with readConcern=sna... Closed
Duplicate
is duplicated by SERVER-41532 Mongos can fail with "a non-retryable... Closed
Related
related to SERVER-47268 noop_createIndexes_not_blocked_by_txn... Closed
related to SERVER-63887 SnapshotUnavailable error on sharded ... Closed
is related to PYTHON-1901 Avoid using readConcern snapshot in s... Closed
is related to SERVER-39624 Reordering of transaction requests fr... Closed
is related to CSHARP-2428 Support mongos pinning for sharded tr... Closed
is related to PHPLIB-677 Duplicate logic in unified test runne... Backlog
is related to RUBY-1942 Avoid using readConcern snapshot in s... Closed
Assigned Teams:
Cluster Scalability
Sprint: Sharding 2019-03-25
Participants:
Case:
Linked BF Score: 33

 Description   

Because of the problem described in SERVER-39624, mongos's ability to retry transaction statements that fail on stale version and snapshot errors without aborting the overall sharded transaction was disabled in favor of returning a TransientTransactionError label, so the client retries the entire transaction with a higher transaction number.

This behavior could be problematic for long running transactions in particular because a stale version error on a late statement would force the entire transaction to be retried, instead of just the failed statement. The purpose of this ticket is to come up with a safe way to allow mongos to retry on these errors within a transaction that addresses the problem from SERVER-39624.



 Comments   
Comment by Karen Yau [ 09/Oct/23 ]

Hi team, any ETA of this fix? Thanks.

 

Comment by Jack Mulrow [ 20/Feb/19 ]

Copying my proposed approach from the linked ticket:

Since the fundamental problem is that mongos can send multiple requests to the same shard with startTransaction=true for a txnId, we might be able to fix this by adding another field to the txnId that is generated by mongos and transparent to the user. Then each time mongos sends a shard startTransaction=true, it can generate a new value for this field (transaction version?) that shards can use to distinguish the earlier attempts. This field could be scoped to a particular txnNumber, so the existing machinery for client retries on transient transaction errors would still work, i.e. any comparison of txnIds always compares the txnNumber before this field.

If we include each shard's expected version in the participant list and send it with prepare/commit (or every request within a transaction), we can abort the entire transaction if any shard has an unexpected version and return a transient transaction error label so the client can retry with a higher txnNumber, since this should only happen because of reordered messages if the client operates correctly.

We could even get rid of the aborts between statement retries by making shards treat a higher transaction version the same as a higher txnNumber and overwrite any state from a previous attempt.

Generated at Thu Feb 08 04:52:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.