[DOCS-585] FAQ: Sharding with Large Number of Small Collections Created: 07/Oct/12  Updated: 30/Oct/23  Resolved: 18/May/16

Status: Closed
Project: Documentation
Component/s: manual
Affects Version/s: None
Fix Version/s: Server_Docs_20231030

Type: Improvement Priority: Minor - P4
Reporter: A. Jesse Jiryu Davis Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by DOCS-761 Best Practices on Sharding Large Coll... Closed
Participants:
Days since reply: 7 years, 39 weeks, 1 day ago

 Description   

Sharding doesn't make sense data with this kind of model.

User had interesting case of sharded cluster, two shards, single database with huge number of small collections. Shard key was random, non-increasing. "Primary shard" for this database was getting overloaded, balancer wasn't moving chunks off. Cause is obvious: each collection only had one or two chunks, so balancer never considered the collection imbalanced. The cluster as a whole was tragically imbalanced.

Solution: pre-split and move chunks for half the collections. Or create a second database, movePrimary it to the other shard, and create half the new collections on this second database so their data starts on the second shard as their default home.

Better solution: don't create thousands of small collections on a sharded cluster, ensure each collection is large enough to get balanced – more than 20-30 chunks at least.



 Comments   
Comment by Kay Kim (Inactive) [ 18/May/16 ]

We're in the process of rewriting the sharded clusters section, highlighting the complexity introduced with sharded clusters, so people should make sure before they use sharded clusters as well as the need for good shard key. If we need to add further warning, we can but would like to see if the new section (in CR) would cover it.

Generated at Thu Feb 08 07:39:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.