[SERVER-58736] Avoid quadratic behavior in rollback with many collections Created: 21/Jul/21  Updated: 29/Oct/23  Resolved: 05/Nov/21

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 5.0.0
Fix Version/s: 5.2.0, 5.0.5, 5.1.1

Type: Improvement Priority: Major - P3
Reporter: Gregory Wlodarek Assignee: Henrik Edin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
is related to SERVER-59065 CatalogStats uses unsafe CollectionCa... Closed
is related to SERVER-59591 LockerNoop::isW() always returns true... Closed
is related to SERVER-59819 collection_catalog_bm setup causes ti... Closed
is related to SERVER-60462 Global MODE_IS lock is required to ac... Closed
is related to SERVER-60460 Add invariant to check that global lo... Closed
is related to SERVER-62618 Avoid quadratic behavior at startup w... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.1, v5.0
Sprint: Execution Team 2021-08-09, Execution Team 2021-11-01, Execution Team 2021-11-15
Participants:
Linked BF Score: 153

 Description   

Copying the CollectionCatalog is expensive. When there are tens of thousands of collections, operations like rollback become substantially slower. When the server is rolling back to the stable timestamp, the global lock is held in exclusive mode. We should be able to avoid making multiple copies of the collection catalog.



 Comments   
Comment by Githook User [ 17/Nov/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-58736 Add BatchedCollectionCatalogWriter class

Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance.

Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog.

(cherry picked from commit 67ef8cd1ed3d15cee09de7905f7ba9fedfba176d)
Branch: v5.1
https://github.com/mongodb/mongo/commit/2f41591c75b5ef09e804291f9be9e7fe1c75d4b0

Comment by Githook User [ 09/Nov/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-58736 Add BatchedCollectionCatalogWriter class

Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance.

Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog.

(cherry picked from commit 67ef8cd1ed3d15cee09de7905f7ba9fedfba176d)
Branch: v5.0
https://github.com/mongodb/mongo/commit/45d16ad9a503ee7398894c3b117e6e05262aad33

Comment by Githook User [ 05/Nov/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-58736 Add BatchedCollectionCatalogWriter class

Allows multiple writes to be performed on the CollectionCatalog while reusing a single copy of the CollectionCatalog instance.

Used when opening the catalog to avoid quadratic behavior in the number of Collections in the catalog.
Branch: master
https://github.com/mongodb/mongo/commit/67ef8cd1ed3d15cee09de7905f7ba9fedfba176d

Comment by Gregory Wlodarek [ 06/Oct/21 ]

I've reverted the original fix for this. Performing writes in-place while the exclusive global lock is held in the CollectionCatalog made it unsafe for other callers to call CollectionCatalog::get() without a global lock. Our AutoGetCollection* helpers don't follow this pattern, so to avoid future issues with reading the CollectionCatalog while it's being written to in-place, this change was reverted. There has also been a handful of BFs as a cause of this change.

I spoke with henrik.edin about this, and one solution that's more involved would be to create a CollectionCatalog::write() function that performs operations in batches. But this will need more investigation.

Comment by Githook User [ 06/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Avoid quadratic behavior in rollback with many collections"

This reverts commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a.
Branch: master
https://github.com/mongodb/mongo/commit/ed1a1de3bbf155f9e14ad536f47771b9c8ba2561

Comment by Githook User [ 06/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Add benchmark for CollectionCatalog::write()"

This reverts commit 9ced8a4e161b258ed9090645065f330af65ee20b.
Branch: master
https://github.com/mongodb/mongo/commit/c7739cfe47ea9099188580022a334e335e4b42ef

Comment by Githook User [ 06/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Avoid quadratic behavior in rollback with many collections"

This reverts commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a.
Branch: v5.1
https://github.com/mongodb/mongo/commit/08a1d17ecceb1a7e0ea969c61f1e0f11d672e6c2

Comment by Githook User [ 06/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Add benchmark for CollectionCatalog::write()"

This reverts commit 9ced8a4e161b258ed9090645065f330af65ee20b.
Branch: v5.1
https://github.com/mongodb/mongo/commit/5d0357fb1120639d34505c8b4298fa8c5efd45f7

Comment by Githook User [ 05/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Avoid quadratic behavior in rollback with many collections"

This reverts commit bfd27ee94c2d81b50e339011cba52dcf0a3625e2.
Branch: v5.0
https://github.com/mongodb/mongo/commit/9a35771a2de9db3f10a606d8aed1edd33aba9568

Comment by Githook User [ 05/Oct/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: Revert "SERVER-58736 Add benchmark for CollectionCatalog::write()"

This reverts commit d2229dd0ad45f79ed4ea01bb502f5cc48b232d70.
Branch: v5.0
https://github.com/mongodb/mongo/commit/06dfff7d3b02c48e8dfe5b01ce9bb7c545c4d402

Comment by Githook User [ 20/Sep/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-58736 Add benchmark for CollectionCatalog::write()

(cherry picked from commit 9ced8a4e161b258ed9090645065f330af65ee20b)
Branch: v5.0
https://github.com/mongodb/mongo/commit/d2229dd0ad45f79ed4ea01bb502f5cc48b232d70

Comment by Githook User [ 20/Sep/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-58736 Avoid quadratic behavior in rollback with many collections

(cherry picked from commit 2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a)
Branch: v5.0
https://github.com/mongodb/mongo/commit/bfd27ee94c2d81b50e339011cba52dcf0a3625e2

Comment by Githook User [ 02/Aug/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-58736 Add benchmark for CollectionCatalog::write()
Branch: master
https://github.com/mongodb/mongo/commit/9ced8a4e161b258ed9090645065f330af65ee20b

Comment by Githook User [ 02/Aug/21 ]

Author:

{'name': 'Gregory Wlodarek', 'email': 'gregory.wlodarek@mongodb.com', 'username': 'GWlodarek'}

Message: SERVER-58736 Avoid quadratic behavior in rollback with many collections
Branch: master
https://github.com/mongodb/mongo/commit/2bc550f5f4d47b6ae2ca176acc2865f1f4a8035a

Generated at Thu Feb 08 05:45:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.