[SERVER-81864] Provide an API to obtain a list of necessary config database collections for debugging Created: 04/Oct/23  Updated: 06/Feb/24

Status: Blocked
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Maria van Keulen Assignee: Aitor Esteve Alvarado
Resolution: Unresolved Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by TOOLS-3378 Leverage Sharding API for obtaining c... Waiting (Blocked)
Related
related to TOOLS-3444 Include config.placementHistory to th... Accepted
Assigned Teams:
Sharding EMEA
Sprint: Sharding EMEA 2023-10-30, CAR Team 2023-11-13, CAR Team 2023-11-27, CAR Team 2023-12-11, CAR Team 2023-12-25, CAR Team 2024-01-08, CAR Team 2024-01-22, CAR Team 2024-02-05, CAR Team 2024-02-19
Participants:
Story Points: 3

 Description   

Currently, users who would like to review the contents of the Sharding config database for debugging purposes can output this database via mongodump (see TOOLS-3374 for an example).

The collections that can/cannot be output have historically been tracked in hardcoded lists in the mongodump code (and similarly for the mongorestore code). A more robust solution moving forward is to have a Sharding API for obtaining this list of collections, to avoid unexpected auth errors like those described in the TOOLS ticket above, and also to ensure that only the necessary debugging collections are included. This list should ideally include only collections that are safe for both mongodump and mongorestore.

When connecting to any Server versions that don't have this API, the tools should continue to use hardcoded lists. To avoid auth errors like in TOOLS-3374, it would be great to exclude the collections known to result in auth errors.



 Comments   
Comment by Maria van Keulen [ 10/Oct/23 ]

kaloian.manassiev@mongodb.com I'll defer to max.hirschhorn@mongodb.com and martin.barciela-pires@mongodb.com on the answer to that question, as they are more familiar with this workflow than I.

Comment by Kaloian Manassiev [ 10/Oct/23 ]

Also, maria.vankeulen@mongodb.com, can we please clarify what is involved in "... [reviewing] the contents of the Sharding config database for debugging purposes" ? I am asking because it can mean different things:

  1. Access to the catalog: Reviewing the set of sharded (soon to be all) collections in the cluster and their placement
  2. Access to changes to the catalog: Reviewing the placement and log collections, which might be insufficient to see all catalog changes
  3. Access to the config collections of the cluster (not strictly related to the CSRS)

From reading TOOLS-3374 I suspect it is about 1 and 2, right? If that is the case, providing an API listCatalog or something like that might be more practical, although not necessarily scalable if it is about dumping a lot of data to be able to reconstruct a new (empty) cluster.

Comment by Tommaso Tocci [ 06/Oct/23 ]

max.hirschhorn@mongodb.com pointed out that there are collections in the config database that are:

  • not strictly needed  for debugging
  • contain user data
  • can be arbitrary big

One example is the config.image_collection .

Considering this, I think it makes sense to provide an API owned and maintained by core server team that return a list of collection to include in the config dump for debugging purposes.

Comment by Tommaso Tocci [ 06/Oct/23 ]

I was wondering if we can take a simpler approach for this.

The idea is that the mongodump tool will call listCollection on the config database (as it does today). It will try to dump all collections and if the user doesn't have enough permission to read a specific collection, it will simply skip that collection, emit a warning and continue dumping the other collections for which the user has enough permission.

 maria.vankeulen@mongodb.com do you think the proposal make sense? if not, why?

Generated at Thu Feb 08 06:47:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.