[SERVER-58736] Avoid quadratic behavior in rollback with many collections Created: 21/Jul/21 Updated: 29/Oct/23 Resolved: 05/Nov/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 5.0.0 |
| Fix Version/s: | 5.2.0, 5.0.5, 5.1.1 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Gregory Wlodarek | Assignee: | Henrik Edin |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v5.1, v5.0
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2021-08-09, Execution Team 2021-11-01, Execution Team 2021-11-15 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 153 | ||||||||||||||||||||||||||||||||||||
| Description |
|
Copying the CollectionCatalog is expensive. When there are tens of thousands of collections, operations like rollback become substantially slower. When the server is rolling back to the stable timestamp, the global lock is held in exclusive mode. We should be able to avoid making multiple copies of the collection catalog. |
| Comments |
| Comment by Githook User [ 17/Nov/21 ] |
|
Author: {'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}Message: Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance. Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog. (cherry picked from commit 67ef8cd1ed3d15cee09de7905f7ba9fedfba176d) |
| Comment by Githook User [ 09/Nov/21 ] |
|
Author: {'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}Message: Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance. Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog. (cherry picked from commit 67ef8cd1ed3d15cee09de7905f7ba9fedfba176d) |
| Comment by Githook User [ 05/Nov/21 ] |
|
Author: {'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}Message: Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance. Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog. |
| Comment by Gregory Wlodarek [ 06/Oct/21 ] |
|
I've reverted the original fix for this. Performing writes in-place while the exclusive global lock is held in the CollectionCatalog made it unsafe for other callers to call CollectionCatalog::get() without a global lock. Our AutoGetCollection* helpers don't follow this pattern, so to avoid future issues with reading the CollectionCatalog while it's being written to in-place, this change was reverted. There has also been a handful of BFs as a cause of this change. I spoke with henrik.edin about this, and one solution that's more involved would be to create a CollectionCatalog::write() function that performs operations in batches. But this will need more investigation. |
| Comment by Githook User [ 06/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a. |
| Comment by Githook User [ 06/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit 9ced8a4e161b258ed9090645065f330af65ee20b. |
| Comment by Githook User [ 06/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a. |
| Comment by Githook User [ 06/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit 9ced8a4e161b258ed9090645065f330af65ee20b. |
| Comment by Githook User [ 05/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit bfd27ee94c2d81b50e339011cba52dcf0a3625e2. |
| Comment by Githook User [ 05/Oct/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: Revert " This reverts commit d2229dd0ad45f79ed4ea01bb502f5cc48b232d70. |
| Comment by Githook User [ 20/Sep/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: (cherry picked from commit 9ced8a4e161b258ed9090645065f330af65ee20b) |
| Comment by Githook User [ 20/Sep/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: (cherry picked from commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a) |
| Comment by Githook User [ 02/Aug/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: |
| Comment by Githook User [ 02/Aug/21 ] |
|
Author: {'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}Message: |