TransactionCoordinatorService may stall primary step-up from completing when replica set shard steps down and back up quickly

XMLWordPrintableJSON

    • Sharding NYC
    • Minor Change
    • ALL
    • 5
    • 3
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      All TransactionCoordinators from the previous term when the node was primary must have exited before a node can finish stepping up as primary. The mechanisms for interrupting TransactionCoordinators involves interrupting active OperationContext and shutting down the txn::AsyncWorkScheduler's TaskExecutor. However the TransactionCoordinator also waits through the WaitForMajorityService and isn't guaranteed to be interrupted. This results in the node completing its member state PRIMARY transition but being unable to exit "drain mode" where the node can accepts writes as primary.

      One visible symptom of this behavior is for the following message to be logged every 5 seconds.

      [js_test:txn_two_phase_commit_basic] d20040| {"t":{"$date":"2023-02-10T22:40:06.714+00:00"},"s":"I",  "c":"TXN",      "id":22442,   "ctx":"OplogApplier-0","msg":"After 5 seconds of wait there are still sessions left with active coordinators which have not yet completed","attr":{"numSessionsLeft":1}}
      

            Assignee:
            David Chen
            Reporter:
            Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: