Shard key updates can trigger invariant at mongos shutdown

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Replication
    • 200
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Updating a document's shard key value involves a shard targeted by an update returning a WouldChangeOwningShard error when it detects a different shard would own the updated document. Mongos catches this error and will spawn a transaction to atomically delete the document from the old shard and reinsert it on the new shard owner.

      At shutdown, because of the assertion from SERVER-77667 preventing starting a new transaction during shutdown, the shard key update code may fail to start the transaction but still try to abort it, which triggers this invariant if the transaction would have been the first on its logical session. This only happens if the original update was not already in a transaction.

      The specific sequence is:

      1. Mongos catches WouldChangeOwningShard and calls handleWouldChangeOwningShardErrorRetryableWriteLegacy()
      2. It tries to begin a transaction here
      3. This fails because a concurrent shutdown has disallowed starting new transactions, triggering this uassert before initializing any transaction state
      4. That error is caught and mongos attempts to implicitly abort the transaction which fails this invariant because the transaction is still uninitialized

      A quick fix is to extend this check before implicitly aborting to also check if the transaction has been initialized but longer term we may want to refactor TransactionRouter lifetime so uninitialized isn't a special case or move the assertion from SERVER-77667.

            Assignee:
            Unassigned
            Reporter:
            Jack Mulrow
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: