[SERVER-60267] Change jumbo chunk size threshold Created: 28/Sep/21  Updated: 29/Oct/23  Resolved: 05/Oct/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: Pierlauro Sciarelli
Resolution: Fixed Votes: 0
Labels: PM-2321-Chunk-Splitter
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Participants:

 Description   

Review the logic that is used to mark chunks as jumbo to see if it still makes sense after the changes introduced in SERVER-60009 that added the ability to keep chunks greater than maxChunkSize until some controlled threshold.

[EDIT] Chosen solution: mark a chunk as jumbo if its size is at least 2 * maxChunkSize (aka double maxRecsWhenFull).



 Comments   
Comment by Pierlauro Sciarelli [ 09/May/22 ]

Hi jean.coulon@mongodb.com,
the change has also been backported to all currently supported version under the following tickets to backport all the new auto splitter changes (including this) in one shot. Opened DOCS-15329 to document this change (I was not aware the threshold was so precisely documented), thanks for pointing that out.

Comment by Jean Alexandre Coulon [ 09/May/22 ]

Hello pierlauro.sciarelli@mongodb.com

Small question here : is this change supposed to be live in other versions than 5.1.0-rc0

I have a customer in 4.4.13 and it seems that the threshold has been changed from 1.3 to 2. 

From documentation (https://www.mongodb.com/docs/manual/core/sharding-balancer-administration/) :

By default, MongoDB cannot move a chunk if the number of documents in the chunk is greater than 1.3 times the result of dividing the configured chunk size by the average document size. db.collection.stats() includes the avgObjSize field, which represents the average document size in the collection.

From log we see :

Cannot move chunk: the maximum number of documents for a chunk is 618, the maximum chunk size is 67108864, average document size is 217075. Found 3375 documents in chunk

configuredchunksize : 67108864

average document size : 217075

So according to previous formula, maximum number of documents should be equal to :

1,3 * (67108864/217075) = 402

However log told us :

the maximum number of documents for a chunk is 618

So :

2 * (67108864/217075) = 618

Now we are matching the value displayed in log.

According to source code

maxRecsWhenFull = 2 * maxRecsWhenFull; // pad some slack

If this has been changed in all versions, we should change the documentation (DOCSP-22524)

 

Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 05/Oct/21 ]

Author:

{'name': 'Pierlauro Sciarelli', 'email': 'pierlauro.sciarelli@mongodb.com', 'username': 'pierlauro'}

Message: SERVER-60267 Change jumbo chunk size threshold
Branch: master
https://github.com/mongodb/mongo/commit/033e43fe45b29c5d6658b0dc1e1d53b8cc50aecc

Generated at Thu Feb 08 05:49:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.