[SERVER-82883] Recovering TransactionCoordinator on stepup may block acquiring read/write tickets while participants are in the prepared state Created: 07/Nov/23  Updated: 01/Jan/24  Resolved: 12/Dec/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.4.0, 5.0.0, 6.0.0, 7.0.0
Fix Version/s: 7.3.0-rc0, 7.0.5, 6.0.13, 5.0.24, 4.4.28

Type: Bug Priority: Major - P3
Reporter: Jordi Serra Torrens Assignee: Wenqin Ye
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
Related
related to SERVER-60682 TransactionCoordinator may block acqu... Closed
Assigned Teams:
Cluster Scalability
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.0, v6.0, v5.0, v4.4
Sprint: Cluster Scalability 2023-11-27, Cluster Scalability 2023-12-11, Cluster Scalability 2023-12-25
Participants:
Linked BF Score: 155

 Description   

Consider a TransactionCoordinator that has sent the prepare command to the participants and then crashes. The new primary, on stepup, will resume the coordination. There are several points at which this can stall behind a read/write ticket acquisition. This is undesirable, both for performance and because it can cause deadlocks.

Ticket acquisitions occur at:
(1) When TransactionCoordinatorService::onStepUp calls replClientInfo.setLastOpToSystemLastOpTime, which takes the GlobalLock in MODE_IX.
(2) When TransactionCoordinatorService::onStepUp reads config.transaction_coordinators.
(3) When waiting for durable VectorClock. This sometimes results in a write (the first time after stepup, or upon topology changes).
(4) When (re-)persisting the participants list. Note that even though it had already been persisted, if the coordinator had not persisted the decision yet, on recovery we will persist again the participant list. As a separate improvement. we should also consider not doing this write again.

SERVER-60682 made persisting the decision skip ticket acquisition, but did not address these other situations that occur on recovery.

In addition to not skipping ticket acquisition, (1) and (3) do not skip FlowControl either.



 Comments   
Comment by Githook User [ 27/Dec/23 ]

Author:

{'name': 'Wenqin Ye', 'email': 'wenqin908@gmail.com', 'username': 'wenqinYe'}

Message: SERVER-82883 Recovering TransactionCoordinator on stepup should skip acquiring read/write tickets while participants are in the prepared state

GitOrigin-RevId: 24a5efab312e5f8c8253e07e05910fd0ed93e1a2
Branch: v5.0
https://github.com/mongodb/mongo/commit/181bf2c5a5d3c6f6e918011c414eb636ce8f891e

Comment by Githook User [ 26/Dec/23 ]

Author:

{'name': 'Wenqin Ye', 'email': 'wenqin908@gmail.com', 'username': 'wenqinYe'}

Message: SERVER-82883: Recovering TransactionCoordinator on stepup should skip acquiring read/write tickets while participants are in the prepared state

GitOrigin-RevId: e75ba7014caabc5c0a2296fa5105387ecd6c51c6
Branch: v4.4
https://github.com/mongodb/mongo/commit/1b595afaed5fc4ede00a8146900d08801add5f4d

Comment by Githook User [ 15/Dec/23 ]

Author:

{'name': 'Wenqin Ye', 'email': 'wenqin908@gmail.com', 'username': 'wenqinYe'}

Message: SERVER-82883: Recovering TransactionCoordinator on stepup should skip acquiring read/write tickets while participants are in the prepared state

GitOrigin-RevId: b17eab4d2a58b8486cdb357398c33c0108f404cc
Branch: v7.0
https://github.com/mongodb/mongo/commit/ab14f5dbbadbc26d1ba6ba036844757bafa1dcc4

Comment by Githook User [ 15/Dec/23 ]

Author:

{'name': 'Wenqin Ye', 'email': 'wenqin908@gmail.com', 'username': 'wenqinYe'}

Message: SERVER-82883: Recovering TransactionCoordinator on stepup should skip acquiring read/write tickets while participants are in the prepared state

GitOrigin-RevId: 125838b48ea42921da92e99a550faefc895a1755
Branch: v6.0
https://github.com/mongodb/mongo/commit/a062dc737b748e78f5d3cacca16bfd558ac2f1c3

Comment by Githook User [ 11/Dec/23 ]

Author:

{'name': 'Wenqin Ye', 'email': 'wenqin908@gmail.com', 'username': 'wenqinYe'}

Message: SERVER-82883: Recovering TransactionCoordinator on stepup should skip acquiring read/write tickets while participants are in the prepared state

GitOrigin-RevId: fe92af8bb9da945673722a7a4115401cfa95a6ff
Branch: master
https://github.com/mongodb/mongo/commit/f4b555a1e2746c97dc73537c470b62c5833005d2

Comment by Josef Ahmad [ 08/Nov/23 ]

I know little about the transaction coordinator, but assuming that the majority of the work it does is on the critical path for retiring prepared transactions, would it make sense to exempt the coordinator from acquiring tickets altogether? It could be a more sustainable approach than patching up individual storage accesses.

Generated at Thu Feb 08 06:50:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.