Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-77667

Prevent mongos from starting new transactions at shutdown

    • Cluster Scalability
    • Fully Compatible
    • v7.2
    • Cluster Scalability 2024-1-8, Cluster Scalability 2024-1-22
    • 135
    • 3

      At shutdown, the mongos process performs a best-effort attempt to abort any transactions the mongos process may have started. This is beneficial for freeing up transaction resources more quickly because the client/driver must retry any in-progress transactions which haven't had their commit coordination handed off already. This is because the transaction protocol does not support committing a multi-statement transaction through a different mongos from the mongos which originally ran the read/write operations. The different mongos can only be used to recover the original commit xor abort decision for the transaction.

      The implicitlyAbortAllTransactions() function which performs this best-effort attempt to abort any transactions the mongos process may have started. However it doesn't prevent TransactionRouter from being used by a not-yet-interrupted OperationContext and starting a new transaction on a shard. Ordinarily this would be an issue because mongos shutting down is rare and the transaction would eventually be aborted on the shard after the transactionLifetimeLimitSeconds (= 60 seconds by default). In testing the transactionLifetimeLimitSeconds server parameter is set to 24 hours to catch cases where a transaction is unintentionally "leaked" by the system. While the system has liveness through the PeriodicThreadToAbortExpiredTransactions job, a stall would be undesirable to have happen in production.

      One place in testing we've seen show up where new transactions are being started while the mongos process is shutting down is with the ClusterServerParameterRefresher thread reading from the config server primary in a multi-statement transaction. The MODE_IX lock held on the config server primary prevents the testing infrastructure from running its data consistency checks before shutting down the config server replica set. One idea to improve implicitlyAbortAllTransactions() is to set a flag on the SessionCatalog indicating process shutdown has begun. TransactionRouter instances which are obtained from the SessionCatalog can check whether this flag has been set and throw an InterruptedAtShutdown or equivalent error to prevent the mongos process from starting any new transactions.

            Assignee:
            wenqin.ye@mongodb.com Wenqin Ye
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: