Currently, ShardRegistry gets reloaded in the config server while running removeShard:

https://github.com/mongodb/mongo/blob/ec9a2f13d82f141d8aca9e3df9e9112b722f2563/src/mongo/db/s/config/sharding_catalog_manager_shard_operations.cpp#L810
https://github.com/mongodb/mongo/blob/ec9a2f13d82f141d8aca9e3df9e9112b722f2563/src/mongo/db/s/config/sharding_catalog_manager_shard_operations.cpp#L852

However, the reload is done with read concern majority so it might not see the changes that were applied during the removeShard command. Investigate whether if reload is even needed in the config server and if yes, how to fix it.