Details
-
Task
-
Resolution: Done
-
Major - P3
-
None
-
None
-
None
-
None
-
Execution Team 2021-12-13, Execution Team 2021-12-27, Execution Team 2022-01-10, Execution Team 2022-01-24
Description
The fine-grained locking hierarchy currently used in the BucketCatalog is difficult to reason about and has led to several bugs. We are hoping to simplify this without sacrificing too many of the performance gains we achieved by implementing the fine-grained locking in the first place.
We can consider any simplifications, but one might be worth checking first: shard the catalog by bucket metadata. Each shard can be protected by a simple mutex. Much of the work on the insert path could be done outside these mutexes, and operations on unrelated buckets would be unlikely to block each other.