[SERVER-58805] InternalError: DatabaseVersion doesn't exist in database entry { _id: "tap", partitioned: false, primary: "int-gws1" } despite the config server being in binary version 4.2 or later. Created: 23/Jul/21  Updated: 27/Oct/23  Resolved: 30/Jul/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: sandip Divekar Assignee: Antonio Fuschetto
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-58909 Missing versions for "admin" and "con... Closed
Operating System: ALL
Participants:

 Description   

Hi,

We are using sharded cluster in our staging environment. Currently we are at version 4.0. We have upgraded from v3.6 to v4.0 successfully. But now facing issue while upgrade from v4.0 to v4.2. Following is the sequence of upgraded components.

1. Config servers - upgraded to v4.2 - No errors
2. Shards servers - upgraded to v4.2 - No errors
3. Mongos - upgraded to v4.2 with errors

When we upgrade the first mongos server to v4.2 and start the service, we will see below logs in mongos.log file. Also, further if we run db command like “show ** collections”, we will get same error on mongo shell.

*Mongos.log file:*
2021-07-23T11:16:08.823+0000 I  SH_REFR  [ConfigServerCatalogCacheLoader-0] Refresh for database tap took 2 ms and failed :: caused by :: InternalError: DatabaseVersion doesn't exist in database entry { _id: "tap", partitioned: false, primary: "int-gws1" } despite the config server being in binary version 4.2 or later.

**

Mongos shell:

mongos> use tap
switched to db tap
mongos> show collections
2021-07-23T11:16:08.825+0000 E  QUERY    [js] uncaught exception: Error: listCollections failed: {
        "ok" : 0,
        "errmsg" : "DatabaseVersion doesn't exist in database entry { _id: \"tap\", partitioned: false, primary: \"int-gws1\" } despite the config server being in binary version 4.2 or later.",
        "code" : 1,
        "codeName" : "InternalError",
        "operationTime" : Timestamp(1627038991, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1627038991, 1),
                "signature" : {
                        "hash" : BinData(0,"YseuzguVafN0/tLEPsnOSkBM/XY="),
                        "keyId" : NumberLong("6963542575651225631")
                }
        }
} :
_getErrorWithCode@src/mongo/shell/utils.js:25:13
DB.prototype._getCollectionInfosCommand@src/mongo/shell/db.js:835:15
DB.prototype.getCollectionInfos@src/mongo/shell/db.js:883:16
shellHelper.show@src/mongo/shell/utils.js:893:9
shellHelper@src/mongo/shell/utils.js:790:15
@(shellhelp2):1:1
 


    • We also ignored this error and upgraded all mongos servers to v4.2 and restarted the mongos services, but error still exists in mongos.log file. Could you please help us to resolve this issue.


 Comments   
Comment by Antonio Fuschetto [ 30/Jul/21 ]

Hi Sandip,

Thank you for your reply and for the problem description. We can then close this ticket.

Best,
Antonio

Comment by sandip Divekar [ 30/Jul/21 ]

Hi Antonio,

No, there are no issues in Mongo packages while setting FCV to 4.0. We had invalid names in config.databases collection, because of that we were not able to set FCV to 4.0, so we did backup and restore of config data and set fcv to 4.0 which was not correct way, This allowed us to set fcv to 4.0 but version field was missing in config data. To generate version field, correct way was to delete these invalid database names using GUI tool, as we were not able to delete it using mongo shell and then set fcv to 4.0. 

We had faced below issue in setting fcv to 4.0 because of these invalid database names:
https://jira.mongodb.org/browse/SERVER-57026

Ex. of invalid database names:

{ "_id" : "���\u0016", "partitioned" : false, "primary" : "int-gws1" } { "_id" : "��Y�*+", "partitioned" : false, "primary" : "int-gws1" }

 

Thanks,
Sandip.

Comment by Antonio Fuschetto [ 29/Jul/21 ]

Hi Sandip,

Thank you very much for your update on this problem.

As you correctly assumed, by setting FCV to 4.0, a value for the version field is generated for each database stored in the config.databases collections in the Config Server(s). This is a prerequisite for the correct functioning of MongoDB version 4.2.

You mentioned that you faced an issue updating the FCV to 4.0. Was it something related to our product that we can improve?

Many thanks,
Antonio

Comment by sandip Divekar [ 29/Jul/21 ]

Hi Antonio,

You can close this ticket, we have resolved this issue. The cause of this issue was absence of "version" field in config.databases collection in config servers data. 

Invalid entry: { "_id" : "tap", "partitioned" : false, "primary" : "int-gws1" }
Valid entry: { "_id" : "tap", "partitioned" : false, "primary" : "int-gws1", "version" :

{ "uuid" : UUID("fcdf7bd1-17d7-498e-91b0-b359e093292c"), "lastMod" : 1 }

}

Earlier when we migrated from 3.6 to 4.0, we had faced issue while setting fcv to 4.0 which has caused absence of "version" field in config.databases collection.
To resolve this issue,

  • Stopped all mongoses(v4.2)
  • Stopped shards services from all servers(v4.2)
  • Downgraded config servers to v4.0 and restored config servers data from backup. This restored data had fcv set to 3.6 already. For safer side, we had created a separate data directory and then restored data into it.
  • Further we set fcv to 4.0, because of which "version" field got automatically created inside config.databases collection.
  • Further we upgraded config servers back to 4.2 and started shards servers and mongoses.
  • At last, when we upgraded our all mongos servers, the error was gone from mongos.log file.

Thanks for the support and taking this ticket into consideration.

 Thanks,
Sandip.

Generated at Thu Feb 08 05:45:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.