[SERVER-57309] Investigate update chunks performance during setFCV 4.4 «--» 5.0 Created: 31/May/21  Updated: 07/Jun/21  Resolved: 07/Jun/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: Pierlauro Sciarelli
Resolution: Done Votes: 0
Labels: PM-1965-Cleanup
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Sprint: Sharding EMEA 2021-06-14
Participants:

 Description   

Given the multi-update performance regression showed in SERVER-56509 , investigate the performance of chunks update as part of setFCV v4.4 «--» v5.0 and eventual fixes in case slowness is detected.



 Comments   
Comment by Pierlauro Sciarelli [ 07/Jun/21 ]

Closing this investigation ticket and opening SERVER-57487 summarizing the performance problem.

Comment by Pierlauro Sciarelli [ 03/Jun/21 ]

Further notes: sys-perf machines were showing a speed up of ~4x compared to the workstation during testing for SERVER-57307, so the results reported in the table above may be considered as worst case scenario on decent machines.

All the mentioned updates are happening under the chunks lock, meaning that merge/move/split operations are blocked. A possible quick improvement to reduce the window in which balancing is not allowed would be to perform the $unset operations outside such lock.

Comment by Pierlauro Sciarelli [ 02/Jun/21 ]

When updating config.chunk entries during setFCV, the scaling is linear.

Following, the results of some simple downgrade/upgrade experiments with different number of chunks (executed on the virtual workstation).

  downgrade downgrade upgrade upgrade
number of chunks $set ns $unset uuid & timestamp $set uuid & timestamp $unset ns
100K 32 s 27 s 22 s 31 s
500K 180 s 164 s 117 s 144 s
1M 324 s 313 s 211 s 271 s

The current order in which indexes are created/dropped on the added/removed fields also turns out to be optimal. For example, the uuid field is added before creating the 3 secondary indexes on it and this ensures that $set operations don't need to update all the indexes. Similarly, on downgrade the ns indexes are dropped before actually removing the ns field from entries.

Generated at Thu Feb 08 05:41:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.