[SERVER-82221] listCollections and listIndexes should include commit-pending namespaces Created: 16/Oct/23  Updated: 10/Jan/24  Resolved: 13/Dec/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.1, 7.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: Josef Ahmad Assignee: David Dominguez Sal
Resolution: Fixed Votes: 0
Labels: car-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-84176 Fix CollectionCatalog::establishConsi... Closed
is related to SERVER-82235 Make logical initial sync use $listCa... Backlog
Assigned Teams:
Catalog and Routing
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.2, v7.0, v6.0, v5.0
Sprint: CAR Team 2023-11-13, CAR Team 2023-11-27, CAR Team 2023-12-11, CAR Team 2023-12-25
Participants:
Linked BF Score: 135
Story Points: 3

 Description   

listCollections and listIndexes do not include commit-pending metadata. This can race with initial sync: the clone phase may fail to detect a collection or index that is pending commit (specifically, the storage transaction has committed, but the catalog changes are yet to be published to the in-mem catalog), while the beginApplyingTimestamp may point to that catalog change because the storage transaction has committed.
 
We have observed a failure where an initial syncing node did not clone a collection that was implicitly created via index creation (two oplog entries, one for collection creation, one for index creation). The beginApplyingTimestamp was set to the optime of the index creation, and listCollections failed to find the collection because it had yet to be published in the in-mem catalog.
 
This issue is unlikely to be observed in the field - it requires a careful combination of implicit collection creation + slow commit handlers + initial sync pointing the beginApplyingTimestamp to the creation optime. It has also probably existed since the dawn of time (I've reproduced on 6.0 onwards).
 
listCollections and listIndexes always read at latest; one solution would be to include commit-pending entries to their output.



 Comments   
Comment by Githook User [ 02/Jan/24 ]

Author:

{'name': 'David Dominguez-Sal', 'email': 'david.dominguez@mongodb.com', 'username': 'david-dominguez-sal'}

Message: SERVER-82221: listCollections requests up to date information from the collection_catalog (backport SERVER-82221 and SERVER-84176)
Branch: v7.2
https://github.com/mongodb/mongo/commit/923456a8c60a3c4e54c767ba589a3947a308fa79

Comment by Githook User [ 02/Jan/24 ]

Author:

{'name': 'David Dominguez-Sal', 'email': 'david.dominguez@mongodb.com', 'username': 'david-dominguez-sal'}

Message: SERVER-82221: listCollections requests up to date information from the collection_catalog' into '10gen/mongo:v7.0' (backport SERVER-82221 and SERVER-84176)

GitOrigin-RevId: f371cb75bc7aeddd5e984fe67574a0f4fe95ecd9
Branch: v7.0
https://github.com/mongodb/mongo/commit/3e5695619baf60875744fb73f7635fe73a738180

Comment by Githook User [ 13/Dec/23 ]

Author:

{'name': 'David Dominguez-Sal', 'email': 'david.dominguez@mongodb.com', 'username': 'david-dominguez-sal'}

Message: SERVER-82221: listCollections requests up to date information from the collection_catalog

GitOrigin-RevId: 2c7f230772ca5b150b209973c1c62942ba1f26e4
Branch: master
https://github.com/mongodb/mongo/commit/405e32eff67c28fcc6b769a7a02f221d43ce2156

Generated at Thu Feb 08 06:48:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.