[SERVER-62700] The rename DDL violates some ShardServerCatalogCacheLoader constraints when the cached metadata collections are UUID-based Created: 18/Jan/22  Updated: 17/Jun/22  Resolved: 24/Jan/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Sergi Mateo Bellido Assignee: Antonio Fuschetto
Resolution: Won't Fix Votes: 0
Labels: SSCCL-BUG
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Sharding EMEA 2022-01-24
Participants:

 Description   

When we enable the long names support, all config.cache.chunks.* collections are UUID-based. This is problematic for the rename DDL operation, let me try to describe the problem:

  • We have a collection "A" with UUID=42 that it is going to be renamed to "B" (let's assume that "B" doesn't exist yet).
  • We acquire the critical section blocking reads/writes over both namespaces.
  • After locally renaming the collection, we commit those changes in the CSRS.
  • Once the operation is committed, we have two namespaces that from the point of view of the ShardServerCatalogCacheLoader refer to the same cached chunks collection: config.cache.chunks.42. This is problematic because despite the fact that we are holding the critical section we could trigger a CatalogCache refresh. Then:
    • We could have a thread refreshing the namespace "A". In this case it will see that the collection was removed (i.e. not present anymore on config.collections and will spawn an async task to drop config.cache.chunks.42.
    • Let's assume that at the same time we have another thread refreshing the namespace "B". This one is going to find it on the CSRS and it is going to check whether config.cache.chunks.42 is created or not, and then it will append the new metadata (if any) to this collection. This is problematic because we don't synchronize the reads that we perform on this collection with the task spawned before. We didn't have this problem before because each namespace had its own and unique cached chunks collection, so it couldn't happen that two different namespaces shared the same cached chunks collection.

I spotted this problem some time ago (SERVER-58465) and thought that because we were holding the critical section this shouldn't be a problem. I was wrong, if the shard executes some "routing code" (i.e. we get the CatalogCache and do some refreshes) we can totally hit it.

rui.liu found this problem in this execution.



 Comments   
Comment by Sergi Mateo Bellido [ 24/Jan/22 ]

Pierlauro and myself discussed about the approach above but wasn't complete.

A solution that would work would be to name the config.cache.chunks.* collections with a hash of the namespace instead of the UUID. We would probably want to persist also this hash in config.cache.collections for post-mortem analysis though.

Comment by Pierlauro Sciarelli [ 21/Jan/22 ]

Would it be an acceptable trick to update the uuid field for the document representing A in the config.cache.collections at some point during the rename? E.g. set it to UUID(0) so that the async drop tries to delete config.cache.chunks.0.

Note that this would not affect the rename operation because:

It seems an easy way to get rid of the race condition with no side effects. I believe it could be done by the DDL coordinator after renaming the metadata on the CSRS and before unblocking CRUDs (around here).

Generated at Thu Feb 08 05:55:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.