Add a discriminator field to ShardingCoordinatorId to allow concurrent sharding DDL coordinators for the same (namespace, operationType)

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Catalog and Routing
    • 3
    • 🟥 DDL
    • None
    • None
    • None
    • None
    • None
    • None

      Sharding coordinators of the same type can either join or conflict, but they do not serialize. Since chunk operations historically serialized with them, we had to add a retry loop in shardsvr_split_chunk_command.cpp that waits for an in-progress conflicting split coordinator to complete and then retries, instead of immediately surfacing ConflictingOperationInProgress to the user.

      The goal of this ticket is to be able to concurrently execute multiple sharding coordinator instances of the same type by adding an extra field to the ShardingCoordinatorId.

      -----------------------------------------------------------------------------------------------------

      Context

      PrimaryOnlyService deduplicates instances by the _id field of the state document. For sharding DDL coordinators, that _id is ShardingCoordinatorId, defined in src/mongo/db/global_catalog/ddl/sharding_coordinator.idl as:

        ShardingCoordinatorId:
            fields:
                namespace:     namespacestring
                operationType: CoordinatorType
      

      The consequence is that at most one coordinator instance can exist at a time for a given (namespace, operationType) pair. When a second request arrives while a coordinator with the same _id is already running, getOrCreateInstance does not queue or wait — if the parameters differ from the in-flight instance it immediately fails the new request with ConflictingOperationInProgress; if the parameters match, it returns the existing instance so the caller piggy-backs on it.

      For coordinators that represent a single logical operation against a collection (e.g. createCollection, dropCollection) this is exactly the desired behaviour. However, some operations are legitimately parallel against the same namespace — multiple requests should be allowed to run concurrently as independent coordinator instances. Two known cases:

      • mergeChunks — different requests target different contiguous ranges of the same collection.
      • dropIndexes — different requests can target different indexes of the same collection.

      Proposed fix

      Add an optional discriminator field to ShardingCoordinatorId in sharding_coordinator.idl:

        ShardingCoordinatorId:
            fields:
                namespace:     namespacestring
                operationType: CoordinatorType
                discriminator: object   # optional
      

      Coordinators that represent a single logical operation per (namespace, operationType) leave discriminator unset — behaviour is unchanged. Coordinators that legitimately run in parallel on the same namespace populate it with a value that uniquely identifies the sub-operation, for example:

      • mergeChunks: (min, max) of the range being merged.
      • dropIndexes: a representation of the index spec / pattern being dropped.

       

      Once this lands:

      • Remove the retry-on-conflict loop in shardsvr_merge/split_chunks_command.cpp and shardsvr_drop_indexes_command.cpp.
      • Stop passing checkOptions = false in those call sites.
      • Simplify the per-coordinator checkIfOptionsConflict overrides (they only need to reject genuinely conflicting parameters for the same sub-operation).

      Considerations

      • The change touches the shared ShardingCoordinatorId IDL, so it affects all sharding DDL coordinators. Since the new field is optional and unset by default, existing coordinators keep their current identity.
      • Backward compatibility for on-disk state documents in config.shardingCoordinators needs to be handled: recovering pre-upgrade documents should continue to parse (strict: false already helps) and should behave as if discriminator were unset.

            Assignee:
            Unassigned
            Reporter:
            Silvia Surroca
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: