|
The consistency checker performs checks even on dropped collections (this is because dropped collections still appear in config.collections with dropped: true). This can lead to noisy NamespaceNotFound in logs on the config server.
original description:
cluster topology:
sharded cluster with 1 config node (primary replica set) , 2 different databearing nodes (primary replica set) and 1 router.
We upgraded our mongo cluster from 4.0.9 to 4.2.8 and a couple of days later to 4.4.0
In the config logs we see every 10 minutes like below (attached mongod log of config).
The databases mentioned are supposed to be dropped from the data bearing nodes.
I'm finding entries in config.collections (config) or config.cached.collections (data bearing nodes) that reference to already dropped db's.
Also initiated flushrouterconfig but that didn't help.
4.2.8
2020-08-09T03:11:01.807+0200 I SHARDING [PeriodicShardedIndexConsistencyChecker] Checking consistency of sharded collection indexes across the cluster
|
2020-08-09T03:11:01.824+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database acc took 0 ms and failed :: caused by :: NamespaceNotFound: database acc not found
|
2020-08-09T03:11:01.828+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database lukes-test took 0 ms and failed :: caused by :: NamespaceNotFound: database lukes-test not found
|
2020-08-09T03:11:01.830+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492084538520 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492084538520 not found
|
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492085358785 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492085358785 not found
|
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492093702966 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492093702966 not found
|
2020-08-09T03:11:01.831+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_1492094907720 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_1492094907720 not found
|
.
|
.
|
.
|
2020-08-09T03:11:02.874+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-00-46 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-00-46 not found
|
2020-08-09T03:11:02.874+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
|
2020-08-09T03:11:02.874+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
|
2020-08-09T03:11:02.874+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-02-08 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-02-08 not found
|
2020-08-09T03:11:02.875+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
|
2020-08-09T03:11:02.875+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
|
2020-08-09T03:11:02.875+0200 I SH_REFR [ConfigServerCatalogCacheLoader-139] Refresh for database ctr-integration-tests_2020-08-07-00-04-07 took 0 ms and failed :: caused by :: NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found
|
2020-08-09T03:11:02.875+0200 I SHARDING [PeriodicShardedIndexConsistencyChecker] Found 0 collections with inconsistent indexes
|
also seen after an upgrade to 4.4.0 but there is not end message from the Checker that 0 collections are found.
The sharded cluster was succesfully upgraded around 10:40 CEST
faced an issue and use the workaround as mentioned in https://jira.mongodb.org/browse/SERVER-50137
{"t":{"$date":"2020-08-11T03:00:23.147+02:00"},"s":"I", "c":"SHARDING", "id":22049, "ctx":"PeriodicShardedIndexConsistencyChecker","msg":"Checking consistency of sharded collection indexes across the cluster"}
|
{"t":{"$date":"2020-08-11T03:00:23.163+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"acc","durationMillis":0,"error":"NamespaceNotFound: database acc not found"}}
|
{"t":{"$date":"2020-08-11T03:00:23.163+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"at-dev","durationMillis":0,"error":"NamespaceNotFound: database at-dev not found"}}
|
{"t":{"$date":"2020-08-11T03:00:23.166+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-test","durationMillis":0,"error":"NamespaceNotFound: database ctr-test not found"}}
|
{"t":{"$date":"2020-08-11T03:00:23.167+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_1492084538520","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_1492084538520 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:23.167+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_1492085358785","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_1492085358785 not found"}}
|
|
.
|
.
|
.
|
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.234+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-07-00-04-07","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-07-00-04-07 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-10-11-50-03","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-10-11-50-03 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.235+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.236+02:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ConfigServerCatalogCacheLoader-47","msg":"Error refreshing cached database entry","attr":{"db":"ctr-integration-tests_2020-08-11-00-01-39","durationMillis":0,"error":"NamespaceNotFound: database ctr-integration-tests_2020-08-11-00-01-39 not found"}}
|
{"t":{"$date":"2020-08-11T03:00:24.292+02:00"},"s":"I", "c":"SHARDING", "id":20997, "ctx":"conn4315","msg":"Refreshed RWC defaults","attr":{"newDefaults":{}}}
|
|