Hello,

We have a quite large cluster (over 20 shards / multiple databases and collections / config database having over 1M documents in chunk collection) . We've observed impact during setting FCV to 4.0 on such a cluster while the whole cluster "freezes" when cache chunks are refreshed on each shard.

============

My understanding of the setting of FCV in case of 4.0 is:

1. FCV is triggered on mongos : https://github.com/mongodb/mongo/blob/91e3352a1aa717674575fce3cc6edb2f279a4479/src/mongo/s/commands/cluster_set_feature_compatibility_version_cmd.cpp
2. mongos sends the command to config servers: https://github.com/mongodb/mongo/blob/91e3352a1aa717674575fce3cc6edb2f279a4479/src/mongo/s/commands/cluster_set_feature_compatibility_version_cmd.cpp#L94
3. config replicaset executes https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp
4. it does that through the block dedicated to config server: https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L184
5. Within this block config will update to new FCV and trigger upgradeChunksHistory for each collection.
6. It will trigger the upgrade of FCV on each shard: https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L226.
7. setting FCV on config replicaset.
   ===============================

Upgrade was performed from 3.6.10 to 4.0.15.

Is there any way to mitigate the impact of enabling FCV on large cluster with lots of chunks?

Thank you,

Cristian