[SERVER-51108] Performance impact when enabling FCV 4.0 Created: 23/Sep/20  Updated: 03/Nov/20  Resolved: 30/Sep/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.15
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Cristian Radan Assignee: Dmitry Agranat
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-49544 Config server should send setFCV to a... Backlog
Participants:

 Description   

Hello,

We have a quite large cluster (over 20 shards / multiple databases and collections / config database having over 1M documents in chunk collection) . We've observed impact during setting FCV to 4.0 on such a cluster while the whole cluster "freezes" when cache chunks are refreshed on each shard.

============

My understanding of the setting of FCV in case of 4.0 is:

  1. FCV is triggered on mongos : https://github.com/mongodb/mongo/blob/91e3352a1aa717674575fce3cc6edb2f279a4479/src/mongo/s/commands/cluster_set_feature_compatibility_version_cmd.cpp
  2. mongos sends the command to config servers: https://github.com/mongodb/mongo/blob/91e3352a1aa717674575fce3cc6edb2f279a4479/src/mongo/s/commands/cluster_set_feature_compatibility_version_cmd.cpp#L94
  3. config replicaset executes https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp
  4. it does that through the block dedicated to config server: https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L184
  5. Within this block config will update to new FCV and trigger upgradeChunksHistory for each collection.
  6. It will trigger the upgrade of FCV on each shard: https://github.com/mongodb/mongo/blob/v4.0/src/mongo/db/commands/set_feature_compatibility_version_command.cpp#L226.
  7. setting FCV on config replicaset.
    ===============================

Upgrade was performed from 3.6.10 to 4.0.15.

Is there any way to mitigate the impact of enabling FCV on large cluster with lots of chunks?

Thank you,

Cristian

 

 



 Comments   
Comment by Dmitry Agranat [ 30/Sep/20 ]

Hi radan@adobe.com,

Among other improvements, the most relevant ticket for the reported issue is SERVER-49544. I will link your report to SERVER-49544 and resolve as a duplicate. You can watch SERVER-49544 for updates.

Regards,
Dima

Comment by Dmitry Agranat [ 24/Sep/20 ]

Hi radan@adobe.com, thank you for the report.

We do have some improvements in this area but these are aimed at 4.4. I am checking if any of these improvements are aimed to be backported to earlier versions.

Thanks,
Dima

Generated at Thu Feb 08 05:24:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.