[SERVER-52886] Sharded cluster is balancing a lot and creating a lot of very small chunks Created: 16/Nov/20 Updated: 05/Aug/21 Resolved: 15/Jul/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 4.4.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Henri-Maxime Ducoulombier | Assignee: | Eric Sedor |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
We recently (mid-october) migrated our production sharded cluster to 4.4.1 and we are experiencing a strange behavior with the balancer. When we started the new cluster, we calculated the number of initial chunks using the default 64MB chunksize, and we restored the collection after a pre-split operation. So we started with 2700 chunks, hashed shard key, for a collection of 170GB and 16M+ documents. Average document size in collection is 10KiB and consistent across the shards. On our dev/test shard, here is a shard detail:
The problem is that, in production, the balancer has been running like crazy and we now have almost 60K of very small chunks.
Please not that we did NOT change the chunk size, and that the config collection does not include any document with _chunksize. Can anybody please advise on what we should do and where to investigate ? I'm afraid this could lead to performance issues or worse, unavailability in case of hard limits in chunks count or size. |
| Comments |
| Comment by Pierlauro Sciarelli [ 05/Aug/21 ] |
|
Hi hmducoulombier@marketing1by1.com, thank you for detailing so deeply the use case. From your description, it looks like chunks were split always in an unfair way: a chunk of size maxChunkSize / 2 + ε was being split in two chunks left (size a bit less than maxChunkSize / 2) and right (containing very few documents, potentially just one). As a consequence, even just one insert in chunk left would trigger an additional split, same for subsequent inserts on the new left chunk and so on. This way, it's easy to end up in the pathological situation you described. This behavior has been caused by the 2 reasons listed below (both going to be solved under 1) The policy to choose split points (not user-configurable, part of splitVector) Split every N keys, with N being the number of keys estimated to correspond to maxChunkSize / 2. It's a fairly distributive policy in case of chunks being split in a lot of smaller chunks (e.g. after re-enabling the balancer), but a very bad choice in case of chunks being continuously split in 2 (e.g. if the balancer has always been enabled). 2) The hashed shard key Let's say that N is 100.
PS: I am not encouraging in any way to use non-hashed monotonically increasing shard keys as it may affect scalability. This problem is going to be solved soon and anyway it is very rare to end up in the described pathological situation. |
| Comment by Eric Sedor [ 15/Jul/21 ] |
|
Hi hmducoulombier@marketing1by1.com, I wanted to get back to you to let you know I'm going to deduplicate this ticket against High counts of small chunks could be undesirable depending on whether or not the balancer is having trouble keeping up with chunk migration. But by itself it is not usually a cause for concern. That said, it does make sense for us to be ensuring that the system is attempting to make use of up to the max chunk size. In Be well! |
| Comment by Eric Sedor [ 15/Jan/21 ] |
|
Hi hmducoulombier@marketing1by1.com and happy new year. I wanted to see if you saw my last message and can provide that context. Thank you! |
| Comment by Henri-Maxime Ducoulombier [ 24/Nov/20 ] |
|
Hello Eric, Thank you for your reply. I uploaded several files using the secure portal and the CURL command. A mongod config server log (the primary) -> mongod-support1.log Please note, regarding the sh.status() result, that another db in the same cluster has the same issue. Our Mongod logs are all recorded and uploaded to cloudwatch and I can run requests to search for a given pattern if needed. Henri-Maxime |
| Comment by Eric Sedor [ 24/Nov/20 ] |
|
Hi Henri-Maxime Ducoulombier, In general, the best place to start to narrow down an issue about sharding behavior is the MongoDB Developer Community Forums. While a specific bug or needed feature could be involved in any given issue, it often helps to walk through the expected behavior of a sharded cluster, which can vary widely by use-case. That said, 279 docs per chunk does look low to me. Can you upload to this secure upload portal the output of sh.status() and the mongod logs from a specific shard? I'll take a look to see if anything is obviously amiss. Gratefully, |