[SERVER-79661] Make "internalRenameIfOptionsAndIndexesMatch" command sharding-aware Created: 03/Aug/23  Updated: 29/Oct/23  Resolved: 09/Oct/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Task Priority: Major - P3
Reporter: David Storch Assignee: Jordi Serra Torrens
Resolution: Fixed Votes: 0
Labels: pm3229-m1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-79670 Make rename collection compatible wit... Closed
depends on SERVER-81444 Don't use temp:true collections for $... Closed
depends on SERVER-81484 Take DDL lock of view nss on Sharding... Closed
Problem/Incident
Related
related to SERVER-79693 Improve MongoProcessInterface methods... Closed
is related to SERVER-81975 Allow sharded target collection on in... Open
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Sprint: QE 2023-09-04, Sharding EMEA 2023-10-02, Sharding EMEA 2023-10-16
Participants:
Linked BF Score: 135

 Description   

The "internalRenameIfOptionsAndIndexesMatch" is an internal command designed specifically to be used by $out. It renames a collection but fails if the indexes or options of the destination collection which is being overwritten by the rename do not match those of the source collection. $out works by writing the data to a temporary collection and then as a final step uses "internalRenameIfOptionsAndIndexesMatch" to replace the existing collection with the new data.

For $out in sharded clusters, the node executing the $out always targets this command to the primary node of the db primary shard. My understanding is that this targeting behavior is acceptable because the db primary is supposed to coordinate DDL commands. However, the command is not sharding-aware in that it looks like it always performs the rename locally. If either the source or destination collections are unsplittable collections placed on a different shard, then the "internalRenameIfOptionsAndIndexesMatch" becomes a distributed operation. This situation will be possible once we allow unsharded collections to move rather than requiring them to reside on the primary shard.



 Comments   
Comment by Githook User [ 09/Oct/23 ]

Author:

{'name': 'Jordi Serra Torrens', 'email': 'jordi.serra-torrens@mongodb.com', 'username': 'jordist'}

Message: SERVER-79661 Make the 'internalRenameIfOptionsAndIndexesMatch' command sharding-aware
Branch: master
https://github.com/mongodb/mongo/commit/6176c2dbd68adc5094496db09aebd3abf7f960b6

Comment by Jordi Serra Torrens [ 07/Aug/23 ]

Some ideas:
Since internalRenameIfOptionsAndIndexesMatch now needs to be sharding-aware and modify the sharding catalog, it should be implemented by using the RenameCollectionCoordinator. We can either (a) make internalRenameIfOptionsAndIndexesMatch spin up a RenameCollectionCoordinator when in a sharded environment, or (b) make $out issue a _shardsvrRenameCollection command with some extra options about the "options and indexes match". Both are to be executed by the dbPrimary shard.

Either way, RenameCollectionCoordinator needs to be extended to support the "option and indexes match" functionality. Currently `internalRenameIfOptionsAndIndexesMatch` uses the database exclusive lock to ensure stability between checking the indexes/options and executing the (local) rename. On a sharded cluster, when the collection is not located on the primary shard, the DB lock does no longer provide the required guarantees. Instead, the DDL lock does (for the most part).

  • RenameCollectionCoordinator takes and holds DDL lock.
  • Changing collection options (collmod, create) serialize on the DDL lock.
  • Dropping indexes serialize on the DDL lock
  • Creating indexes does not serialize on the DDL lock. (createIndexes is currently not a ShardingDDLCoordinator nor takes the DDL lock)

Unless createIndexes is changed to either take the DDL lock or become a ShardingDDLCoordinator, the "index match" check will be racy with concurrent index creation.

Comment by David Storch [ 03/Aug/23 ]

CC jordi.serra-torrens@mongodb.com ivan.fefer@mongodb.com. Hopefully my description of this task is accurate – let me know if there's anything you'd like to add.

Generated at Thu Feb 08 06:41:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.