Hello Guys,

We have been facing an issue with one of our sharded mongo cluster (community edition). 

Cluster Mongo Version: 4.4.18

NShards: 3

Data nodes per shard: 4 + 1 arbiter

Config nodes: 5

Individual mongos clients for all the service machines (~18)

A client which is read heavy is facing the issue of mongos querying data from only 1 shard instead of all 3 shards. The queried collection is a sharded collection.

We tried to run flushRouterConfig cmd before the scripts process that collection to make sure it refreshes its cache to recognize its a sharded collection. but its intermittent.

For example:
Running flushRouterConfig on database level and that particular collection will generate the following log entry:

{"t":{"$date":"2023-07-17T00:10:02.074+00:00"},"s":"I",  "c":"SH_REFR",  "id":24104,   "ctx":"ConfigServerCatalogCacheLoader-6957","msg":"Refreshed cached collection","attr":{"namespace":"dbname.pe_20230716","newVersion":{"0":{"$timestamp":{"t":1,"i":2}},"1":{"$oid":"64a399837a590637cc1a9994"}},"oldVersion":"","durationMillis":1}}
{"t":{"$date":"2023-07-17T01:30:02.216+00:00"},"s":"I",  "c":"SH_REFR",  "id":24104,   "ctx":"ConfigServerCatalogCacheLoader-6976","msg":"Refreshed cached collection","attr":{"namespace":"dbname.pe_20230716","newVersion":{"0":{"$timestamp":{"t":1,"i":2}},"1":{"$oid":"64a399837a590637cc1a9994"}},"oldVersion":"","durationMillis":5}}
 

However, a log from ctx ConfigServerCatalogCacheLoader saying that that collection is indeed refreshed is expected after these lines which we don't observe most of time (sometime works). For example:

{"t":{"$date":"2023-07-18T04:18:36.325+00:00"},"s":"I",  "c":"SH_REFR",  "id":24104,   "ctx":"ConfigServerCatalogCacheLoader-0","msg":"Refreshed cached collection","attr":{"namespace":"dbname.pe_20230730","newVersion":{"0":{"$timestamp":{"t":1,"i":2}},"1":{"$oid":"64b60e8d7a590637cc95899e"}},"oldVersion":"","durationMillis":2}} 

Our current alternative we resorted to after lots of issues is to restart mongos to force refresh and cache all collections and databases just before beginning of our critical processes.

Could you advise on this?

Thanks & Regards