[SERVER-56002] listIndexes can read partial state from renameCollection Created: 09/Apr/21  Updated: 29/Oct/23  Resolved: 20/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.0-rc0, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Mihai Andrei Assignee: Henrik Edin
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-52713 [testing] Add stepdown/kill/terminate... Closed
Duplicate
is duplicated by SERVER-55519 Investigate missing _id index during ... Closed
Problem/Incident
Related
related to SERVER-56877 insert operations may fail to set ind... Closed
related to SERVER-57083 Coverity analysis defect 120111: Pars... Closed
related to SERVER-57324 dropDatabase should not modify Collec... Closed
related to SERVER-57775 collection metadata for multikey not ... Closed
related to SERVER-57127 Refactor multikey and other state out... Closed
is related to SERVER-55519 Investigate missing _id index during ... Closed
is related to SERVER-56023 listCollections can return empty meta... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Execution Team 2021-05-17, Execution Team 2021-05-31
Participants:
Linked BF Score: 158

 Description   

Essentially, the issue appears to be that listIndexes can read from the DurableCatalog in the middle of a concurrent collection rename (after the original target was dropped, but before the source collection is renamed). This looks to be possible because the listIndexes can read from the DurableCatalog using the same catalogId that the collection rename removes from the DurableCatalog.

Some ideas for how to fix this include making listIndexes no longer lock free as well as reimplementing listIndexes to use the IndexCatalog instead of the DurableCatalog.



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 24/May/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-56002 SERVER-56023 Store Collection metadata in the Collection and reply on the copy-on-write machinery to keep it in sync with the durable catalog.

All updates to the metadata needs to happen through the Collection, moved interfaces from the DurableCatalog to the Collection.
Removed back pointer to Collection in IndexCatalogEntryImpl, interfaces now correctly take a const or non-const Collection. This should make its iterface const-correct to avoid making bugs where the copy-on-write system for Collections are bypassed.

Multikey handle is special as it needs to happen without exclusive access to the Collection. Implemented isolation for the Collection metadata when multikey is changed. It handles multi-doc transactions and is only commited to the Collection instance after the write to the durable catalog successfully commits.

listCollections and listIndexes can now safetly read the metadata cache without needing to read from the durable catalog making them safe to do without Collection level locks.

(cherry picked from commit 11de948b0c50df7d12de09ae0f01e791fc5d70d7)
Branch: v5.0
https://github.com/mongodb/mongo/commit/50547a878aaa67722f69c05b05a35236f8f0def9

Comment by Githook User [ 24/May/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-56002 SERVER-56023 Store Collection metadata in the Collection and reply on the copy-on-write machinery to keep it in sync with the durable catalog.

(cherry picked from commit 15e97c021b4e1feba110f31a563076980794d0a5)
Branch: v5.0
https://github.com/10gen/mongo-enterprise-modules/commit/0637bb99c0e672df3f0710a970ae1bb2aad0a0ed

Comment by Githook User [ 20/May/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-56002 SERVER-56023 Store Collection metadata in the Collection and reply on the copy-on-write machinery to keep it in sync with the durable catalog.

All updates to the metadata needs to happen through the Collection, moved interfaces from the DurableCatalog to the Collection.
Removed back pointer to Collection in IndexCatalogEntryImpl, interfaces now correctly take a const or non-const Collection. This should make its iterface const-correct to avoid making bugs where the copy-on-write system for Collections are bypassed.

Multikey handle is special as it needs to happen without exclusive access to the Collection. Implemented isolation for the Collection metadata when multikey is changed. It handles multi-doc transactions and is only commited to the Collection instance after the write to the durable catalog successfully commits.

listCollections and listIndexes can now safetly read the metadata cache without needing to read from the durable catalog making them safe to do without Collection level locks.
Branch: master
https://github.com/mongodb/mongo/commit/11de948b0c50df7d12de09ae0f01e791fc5d70d7

Comment by Githook User [ 20/May/21 ]

Author:

{'name': 'Henrik Edin', 'email': 'henrik.edin@mongodb.com', 'username': 'henrikedin'}

Message: SERVER-56002 SERVER-56023 Store Collection metadata in the Collection and reply on the copy-on-write machinery to keep it in sync with the durable catalog.
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/15e97c021b4e1feba110f31a563076980794d0a5

Comment by James Wahlin [ 06/May/21 ]

Requesting 4.9 backport as we had a BF on that branch that we believe is caused by this issue.

Generated at Thu Feb 08 05:38:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.