Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-76836

setAllowMigrations is executing remote calls with a session checked out

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 7.1.0-rc0
    • Affects Version/s: None
    • Component/s: Sharding
    • Labels:
    • Fully Compatible
    • ALL
    • Sharding EMEA 2023-05-15
    • 110

      SERVER-73539 added replay protection to the setAllowMigrations command, however, this implies having a session checked out, while a refresh on all shards is happening. In the config shard setup, the following deadlock can occur:

      shardA (also config server)

      1. moveChunk from shardB to shardA.
      2. shardA: Some ddl op calls sharding_ddl_util::stopMigrations. For example, in renameCollection, a session X is attached with the _configsvrSetAllowMigrations it sends out to the config server.
      3. shardA (also config server): session X is checked out while running _configsvrSetAllowMigrations.
      4. shardA: during session migration the destination encounters a session with id X, and tries to check it out, but is blocked because of _configsvrSetAllowMigrations.
      5. shardA: _configsvrSetAllowMigrations calls _flushRoutingTableCacheUpdatesWithWriteConcern to all shards.
      6. shardB: _flushRoutingTableCacheUpdatesWithWriteConcern waits for migration source to finish (via recoverRefresh -> wait for migration abort future)
      7. shardB: as part of abort, it waits for _recvChunkReleaseCritSec to succeed. Since session migration is still ongoing on the destination, it will always return an error. But shardA is stuck because session migration is blocked waiting for _configsvrSetAllowMigrations to release the session.

      We should do something similar to the transaction yielder, that is, yield the session while doing remote (or possible blocking) work.

            marcos.grillo@mongodb.com Marcos José Grillo Ramirez
            marcos.grillo@mongodb.com Marcos José Grillo Ramirez
            0 Vote for this issue
            4 Start watching this issue