[SERVER-19919] Chunks that exceed 250000 docs but are under half chunk size get marked as jumbo Created: 13/Aug/15 Updated: 02/Aug/18 Resolved: 19/Oct/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.6.11, 3.0.5, 3.2.10, 3.5.13 |
| Fix Version/s: | 3.4.11, 3.6.0-rc1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Andrew Ryder (Inactive) | Assignee: | Randolph Tan |
| Resolution: | Done | Votes: | 8 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||
| Backport Requested: |
v3.4
|
||||||||||||||||||||||||||||||||
| Steps To Reproduce: |
|
||||||||||||||||||||||||||||||||
| Sprint: | Sharding 2017-10-23 | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||
| Description |
|
If a chunk that contains more than 250000 documents, but is less than half the chunk size, gets selected for a moveChunk operation, the balancer will mark the chunk as jumbo regardless of the shard key range available or the actual total data size of the chunk (so long as it is less than half the chunk size). The issue can occur repeatedly in the same collection The issue is most likely to occur wherever large numbers of small documents are in use. The requirements are:
The average document size threshold can be calculated for 3.2 or later as:
For 3.0 and earlier the average document size threshold formula is:
Note that if chunkSizeMB is raised, so is the threshold for average document sizes, under which the issue might manifest. In these conditions, whenever a moveChunk occurs by the balancer attempting to bring the cluster into balance (for example, after some other split or a new shard is added for capacity), there is a chance of selecting a chunk whose document count actually exceeds 250000 (the separate mongos didn't collude to auto-split this before the limit was reached). 250000 is the hard-coded document count limit for a chunk before a move is rejected as "chunkTooBig". The balancer responds to such a failure by issuing a split request. The split returns a single split point (because the size is still under half the chunk size) and then Note that in the repro steps, the numInitialChunks set to 1 is not required, although doing so makes the issue occur much sooner thanks to the initial imbalance. WorkaroundUsers can lower the chunksize setting, and then clear the jumbo flags on all affected chunks to have them re-evaluated for splitting. |
| Comments |
| Comment by Githook User [ 09/Nov/17 ] |
|
Author: {'name': 'Randolph Tan', 'username': 'renctan', 'email': 'randolph@10gen.com'}Message: (cherry picked from commit 830fe9d2093b365d51aaafadfecefcdd09fd0ede) |
| Comment by Githook User [ 19/Oct/17 ] |
|
Author: {'email': 'randolph@10gen.com', 'name': 'Randolph Tan', 'username': 'renctan'}Message: |
| Comment by alplatonov [ 13/Jun/17 ] |
|
I have collections with large and small document size, this bug affect us too. So i write automatic chunk split script for collections with small document size, but it resolve issue not good enough and the imbalance is still present. |
| Comment by Andrew Ryder (Inactive) [ 29/Aug/15 ] |
|
Hi Derek, Lowering the chunksize is a persistent workaround, however if you don't want to do that, you can alternatively just clear the jumbo flags periodically (the second part of the stated workaround - once a week perhaps) on the affected collection to have the balancer re-assess those chunks. If those chunks have received more documents in the interim they may have reached the size threshold to be split, if not, they'll merely be marked as jumbo again. Either way, doing this periodically will allow the balancer to make progress without modifying the chunksize setting. I realize this is not a solution, but until the issue can be addressed properly this may suffice as a workaround for you. I hope this helps. Kind regards, |
| Comment by Derek Wilson [ 28/Aug/15 ] |
|
I can't use the work around as it will end up splitting chunks that are appropriately sized in other collections. Average document sizes in different collections can be from really tiny (low tens of bytes) to large (ten-ish kb) with hundreds of millions of docs each. There really isn't a good balance in my case. |