Apparently reconfig command removing a server from catalogShard doesn't update the config.shards.
Repro jstests/sharding/mongos_no_replica_set_refresh.js

The half of the root cause is that the updateReplicaSetOnConfigServer() method is skipping changing the config.shards on the config server because the config server is doing it by itself. The fix will be find where on the config this is supposed to happen and adjust it for the catalog shard behavior.