[SERVER-37886] Remove config server as coordinator crutch from coordinator stepdown targeted tests Created: 01/Nov/18  Updated: 29/Oct/23  Resolved: 17/Apr/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.1.11

Type: Task Priority: Major - P3
Reporter: Matthew Saltz (Inactive) Assignee: Jack Mulrow
Resolution: Fixed Votes: 0
Labels: ShardedTxn:DistributedCommit
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-35872 Reconstruct prepared transactions on ... Closed
depends on SERVER-39036 Stop pinning stable timestamp behind ... Closed
depends on SERVER-40081 Move session checkout to before comma... Closed
is depended on by SERVER-40220 Remove "config server is coordinator"... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2019-03-11, Sharding 2019-04-22
Participants:
Linked BF Score: 11

 Description   

Currently, in order to test coordinator failover, we have an override that causes the router to always use the config server as the coordinator shard for a transaction. Once prepare failover is ready, we should move the coordinator back onto the first shard to be touched by a transaction.



 Comments   
Comment by Githook User [ 17/Apr/19 ]

Author:

{'email': 'jack.mulrow@mongodb.com', 'name': 'Jack Mulrow', 'username': 'jsmulrow'}

Message: SERVER-37886 Remove config server as coordinator crutch from coordinator stepdown targeted tests

This reverts commit 7f620154e595d2c1e6c7af79fc62070ced3bb941.
Branch: master
https://github.com/mongodb/mongo/commit/bb452e17d528733e5e645d814bc237d7a26e6c0b

Comment by Githook User [ 16/Apr/19 ]

Author:

{'email': 'jack.mulrow@mongodb.com', 'name': 'Jack Mulrow', 'username': 'jsmulrow'}

Message: SERVER-37886 Make prepareTransaction set Client last OpTime to the greater of the prepare OpTime and system last OpTime if the Client's last OpTime is less than the prepare OpTime

This reverts commit bc276668e8d992ff833fdbfd0f4280419d11eda1.
Branch: master
https://github.com/mongodb/mongo/commit/19b622ebfb42a525f38e278c09f440eb47b12f1e

Comment by Esha Maharishi (Inactive) [ 18/Mar/19 ]

I also marked this as depends on SERVER-39036, because the server fix to the prepare command above ("Make prepareTransaction set Client last OpTime to the greater of the prepare OpTime and system last OpTime if the Client's last OpTime is less than the prepare OpTime") exacerbated the existing issue that multiple coordinators for the same session can deadlock with each other on a participant, causing the txn_two_phase_commit_failover.js to fail more frequently than it is now.

Comment by Esha Maharishi (Inactive) [ 18/Mar/19 ]

Marking as depends on SERVER-40081, because currently stepdown can deadlock with a command waiting for writeConcern under a checked out session.

The deadlock is a result of a lock order inversion between taking the ReplicationCoordinator mutex and checking out a session:

The deadlock that ensues is either:

1) Stepdown acquires the ReplicationCoordinator mutex before the command calls ReplicationCoordinator::awaitReplication. The command blocks trying to acquire the ReplicationCoordinator mutex in ReplicationCoordinator::awaitReplication, and the stepdown blocks in checkOutSessionForKill.

OR

2) Stepdown acquires the ReplicationCoordinator mutex while the command is in waitForConditionOrInterrupt as part of ReplicationCoordinator::_awaitReplication_inlock. The stepdown successfully marks the command's OperationContext as killed, but the command blocks trying to acquire the ReplicationCoordinator mutex to wake up from its sleep, and the stepdown blocks in checkOutSessionForKill.

I originally thought that SERVER-40069 (ensuring the GlobalLockAcquisitionTracker says that an OperationContext acquired a global lock if the OperationContext ever held a global lock) would prevent the deadlock, because then the stepdown would mark the OperationContext as killed while not holding the ReplicationCoordinator mutex, through KillOpContainer::startKillOpThread. However, this would only prevent the second deadlock described above, not the first, since the command would not check for interrupt until after acquiring the ReplicationCoordinator mutex for awaitReplication.

Comment by Githook User [ 08/Mar/19 ]

Author:

{'name': 'Esha Maharishi', 'username': 'EshaMaharishi', 'email': 'esha.maharishi@mongodb.com'}

Message: Revert "SERVER-37886 Make prepareTransaction set Client last OpTime to the greater of the prepare OpTime and system last OpTime if the Client's last OpTime is less than the prepare OpTime"

This reverts commit 8f3ad3eab4631a393cb4c2bfff69015baf63ebc9.
Branch: master
https://github.com/mongodb/mongo/commit/bc276668e8d992ff833fdbfd0f4280419d11eda1

Comment by Githook User [ 08/Mar/19 ]

Author:

{'name': 'Esha Maharishi', 'email': 'esha.maharishi@mongodb.com', 'username': 'EshaMaharishi'}

Message: Revert "SERVER-37886 Remove config server as coordinator crutch from coordinator stepdown targeted tests"

This reverts commit a0555e9be29e21c1c822d4bfc860c66b6838a00f.
Branch: master
https://github.com/mongodb/mongo/commit/7f620154e595d2c1e6c7af79fc62070ced3bb941

Comment by Githook User [ 08/Mar/19 ]

Author:

{'name': 'Esha Maharishi', 'username': 'EshaMaharishi', 'email': 'esha.maharishi@mongodb.com'}

Message: SERVER-37886 Remove config server as coordinator crutch from coordinator stepdown targeted tests
Branch: master
https://github.com/mongodb/mongo/commit/a0555e9be29e21c1c822d4bfc860c66b6838a00f

Comment by Githook User [ 08/Mar/19 ]

Author:

{'name': 'Esha Maharishi', 'username': 'EshaMaharishi', 'email': 'esha.maharishi@mongodb.com'}

Message: SERVER-37886 Make prepareTransaction set Client last OpTime to the greater of the prepare OpTime and system last OpTime if the Client's last OpTime is less than the prepare OpTime
Branch: master
https://github.com/mongodb/mongo/commit/8f3ad3eab4631a393cb4c2bfff69015baf63ebc9

Comment by Esha Maharishi (Inactive) [ 07/Dec/18 ]

Actually, using the override that retries the transaction entirely would allow the tests to pass while still testing that the coordinator resumed coordinating commit on stepup.

I was worried the override would mean the tests would pass even if the coordinator never resumed coordinating any commits, but that's not true. If the coordinator never resumed coordinating the commit and there were prepared participants, the tests would hang.

Comment by Esha Maharishi (Inactive) [ 07/Dec/18 ]

Since participants abort unprepared transactions on stepdown, we will never be able to do passthrough testing of coordinator failover unless the coordinator is not a participant (the tests running in the passthrough may expect the transaction to commit).

Generated at Thu Feb 08 04:47:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.