[DOCS-585] FAQ: Sharding with Large Number of Small Collections Created: 07/Oct/12 Updated: 30/Oct/23 Resolved: 18/May/16 |
|
| Status: | Closed |
| Project: | Documentation |
| Component/s: | manual |
| Affects Version/s: | None |
| Fix Version/s: | Server_Docs_20231030 |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | A. Jesse Jiryu Davis | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Days since reply: | 7 years, 39 weeks, 1 day ago | ||||||||
| Description |
|
Sharding doesn't make sense data with this kind of model. User had interesting case of sharded cluster, two shards, single database with huge number of small collections. Shard key was random, non-increasing. "Primary shard" for this database was getting overloaded, balancer wasn't moving chunks off. Cause is obvious: each collection only had one or two chunks, so balancer never considered the collection imbalanced. The cluster as a whole was tragically imbalanced. Solution: pre-split and move chunks for half the collections. Or create a second database, movePrimary it to the other shard, and create half the new collections on this second database so their data starts on the second shard as their default home. Better solution: don't create thousands of small collections on a sharded cluster, ensure each collection is large enough to get balanced – more than 20-30 chunks at least. |
| Comments |
| Comment by Kay Kim (Inactive) [ 18/May/16 ] |
|
We're in the process of rewriting the sharded clusters section, highlighting the complexity introduced with sharded clusters, so people should make sure before they use sharded clusters as well as the need for good shard key. If we need to add further warning, we can but would like to see if the new section (in CR) would cover it. |