We found this problem on the CatalogCache but I think it is a generic problem for other components that rely on some metadata stored on the config server.

Note that the problem explained below is just for secondary nodes on the CSRS, everything is fine on the primary node.

Let's assume that for whatever reason a secondary node of the CSRS receives a StaleShardVersion error from a shard, stating that the current shard version is X. That shard knew a configTime of T1, which is also part of the response, but it is not going to be gossiped in on the CSRS. As part of the StaleShardVersion error handling, we mark the routing info (i.e. the catalog cache) for that namespace as stale, expecting that the next time we need it we will perform a refresh. Note that is responsibility of the CSRS nodes to advance the configTime. Thus, it could happen that the secondary node had a configTime of T0 and when it perfoms a refresh it doesn't find the version X:  for config server nodes, the loader doesn't attach an $afterClusterTime.