[SERVER-939] Ability to distribute collections in a single db Created: 28/Sep/09 Updated: 26/Oct/23 |
|
| Status: | Blocked |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Eliot Horowitz (Inactive) | Assignee: | Backlog - Catalog and Routing |
| Resolution: | Unresolved | Votes: | 96 |
| Labels: | oldshardingemea, pm-1051-legacy-tickets | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Assigned Teams: |
Catalog and Routing
|
||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
The implementation of this feature (without moveCollections) has the following work items:
To implement moveCollections some additional work is probably necessary, but we can leverage much of the cloneCollection framework. |
| Comments |
| Comment by Cris Insignares Cuello [ 06/Jul/23 ] |
|
This was part of Global balancer, PM-2325 that was closed as execution it might result in decreased cluster performance for existing customers |
| Comment by Asya Kamsky [ 09/Nov/14 ] |
|
crazyzh1984 first of all, document size limit is 16MB and not 4MB. And you can already distribute collections across shards via tag aware sharding - I have a write-up with some examples here: http://askasya.com/post/taggedcollectionbalancing In addition you can store all blog posts in a single collection, just identifying it with a specific author and then shard the collection by author. |
| Comment by JinYan.Zhang [ 09/Nov/14 ] |
|
Because documents has size limit of 4MB, so we use collections to aggregate records. We wish to shard on collections dynamically, a use case: We really expect this feature. |
| Comment by auto [ 08/Feb/13 ] |
|
Author: {u'date': u'2013-01-29T16:08:11Z', u'email': u'randolph@10gen.com', u'name': u'Randolph Tan'}Message: SERVER-939 abilitiy to distribute collections in a single db Port tests from d_chunk_manager_tests.cpp for collection_manager_test.cpp |
| Comment by auto [ 22/Jan/13 ] |
|
Author: {u'date': u'2013-01-22T18:36:24Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Removed unused default parameter from extract function |
| Comment by auto [ 22/Jan/13 ] |
|
Author: {u'date': u'2013-01-22T17:33:03Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Made source argument of parseBSON const ref |
| Comment by auto [ 22/Jan/13 ] |
|
Author: {u'date': u'2013-01-22T17:26:31Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Change FIELD_VALID to FIELD_SET |
| Comment by auto [ 22/Jan/13 ] |
|
Author: {u'date': u'2013-01-22T17:23:50Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Made is*Set methods const |
| Comment by auto [ 18/Jan/13 ] |
|
Author: {u'date': u'2013-01-18T22:46:27Z', u'email': u'randolph@10gen.com', u'name': u'Randolph Tan'}Message: SERVER-939 abilitiy to distribute collections in a single db Fixed test failure due to uninitialized flag in the mock cursor. |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T17:04:05Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added defaults to fields to fix field_parser test |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:27Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update ChangelogType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:28Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update ChunkType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:28Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update CollectionType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:28Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update VersionType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:29Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update LockpingsType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:29Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update LocksType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:30Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update MongosType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:30Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update SettingsType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:30Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update ShardType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T11:50:31Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update TagsType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-17T12:13:21Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Update DatabaseType with new method of recording field presence and handling defaults |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-04T19:46:14Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Store defaults in BSONField and handle defaults in FieldParser extract functions |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2012-12-19T17:26:41Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added information about whether field was set to field_parser |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-10T20:11:30Z', u'email': u'randolph@10gen.com', u'name': u'Randolph Tan'}Message: SERVER-939 abilitiy to distribute collections in a single db Fix memory leak when error is encountered while creating a CollectionManager |
| Comment by auto [ 17/Jan/13 ] |
|
Author: {u'date': u'2013-01-10T20:02:37Z', u'email': u'randolph@10gen.com', u'name': u'Randolph Tan'}Message: SERVER-939 abilitiy to distribute collections in a single db Port tests from d_chunk_manager_tests.cpp for metadata_loader_test.cpp |
| Comment by auto [ 19/Dec/12 ] |
|
Author: {u'date': u'2012-12-14T20:05:24Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added errMsg to parseBSON in *Type classes |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-05T23:40:17Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed ChangelogFields to ChangelogType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-05T22:52:52Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.changelog collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T01:32:04Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed LockPingFields to LockpingsType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T01:25:57Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.lockpings collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-05T21:24:35Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed MongosFields to MongosType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T20:04:40Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed TagFields to TagsType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T19:52:51Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.tags collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T01:04:06Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Changed LockFields to LocksType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T01:00:12Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.locks collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-07T02:19:28Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed SettingsFields to SettingsType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-07T02:05:38Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.settings collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-01T01:34:37Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed DatabaseFields to DatabaseType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-02T01:52:01Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Added type class for documents in config.mongos collection |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-12-01T01:08:14Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed CollectionFields to CollectionType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-11-30T19:47:49Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed ShardFields to ShardType |
| Comment by auto [ 18/Dec/12 ] |
|
Author: {u'date': u'2012-11-30T02:16:08Z', u'name': u'Shaun Verch', u'email': u'shaun.verch@10gen.com'}Message: SERVER-939 Changed ChunkFields to ChunkType |
| Comment by auto [ 11/Dec/12 ] |
|
Author: {u'date': u'2012-12-11T19:29:01Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 fixes to database type |
| Comment by auto [ 11/Dec/12 ] |
|
Author: {u'date': u'2012-12-11T19:28:45Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 field parser and BSON enhancements for vectors and maps |
| Comment by auto [ 06/Dec/12 ] |
|
Author: {u'date': u'2012-12-06T15:43:28Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 temporarily remove scoped dist lock tests, currently incompatible with debug builds |
| Comment by Greg Studer [ 05/Dec/12 ] |
|
Yeah - forgot that debug builds fail immediately on verify. Will work to try and fix. |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-12-03T21:38:28Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 rollback bson element error message change |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-12-03T20:50:15Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 minor fixes to test cases for new type_ parsing |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-26T19:11:46Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 make scopedDbConnection take ConnectionStrings directly |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-30T21:32:43Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 enhance owned pointer vector and add owned pointer map |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-26T19:14:35Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 modify config types for config version changes |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-26T19:14:51Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 error handling for string* |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-26T19:09:49Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 toString() for random part of OID |
| Comment by auto [ 03/Dec/12 ] |
|
Author: {u'date': u'2012-11-26T19:09:25Z', u'email': u'greg@10gen.com', u'name': u'Greg Studer'}Message: SERVER-939 simplify use of scoped distributed and balancer lock |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-29T22:05:23Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added shard tags to field constants and document type |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-26T23:22:02Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added field type constants for config.settings collection |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-29T10:17:59Z', u'email': u'tad@10gen.com', u'name': u'Tad Marshall'}Message: SERVER-939 fix compile |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-29T10:10:19Z', u'email': u'tad@10gen.com', u'name': u'Tad Marshall'}Message: SERVER-939 whitespace (indent) for cpplint |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-29T06:04:38Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Cleaned CollectionManager. |
| Comment by auto [ 29/Nov/12 ] |
|
Author: {u'date': u'2012-11-26T20:15:08Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 new CollectionManager and MetadataLoader classes (not yet hooked). |
| Comment by auto [ 22/Nov/12 ] |
|
Author: {u'date': u'2012-11-21T19:38:44Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added field type constants for config.changelog collection |
| Comment by auto [ 21/Nov/12 ] |
|
Author: {u'date': u'2012-11-20T21:07:28Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added field type constants for config.mongos collection |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-20T02:57:50Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Introduced type for config.shards collection. |
| Comment by auto [ 20/Nov/12 ] |
|
Author: {u'date': u'2012-11-19T20:29:23Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added field type constants for config.lockpings collection |
| Comment by auto [ 19/Nov/12 ] |
|
Author: {u'date': u'2012-11-19T23:49:17Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Introduced types for config.databases and config.chunks collections. |
| Comment by auto [ 19/Nov/12 ] |
|
Author: {u'date': u'2012-11-13T19:58:20Z', u'email': u'shaun.verch@10gen.com', u'name': u'Shaun Verch'}Message: SERVER-939 Added field type constants for config.locks collection |
| Comment by auto [ 11/Nov/12 ] |
|
Author: {u'date': u'2012-11-11T00:10:46Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Changed FieldParser to have the correct BSON ownership semantics. |
| Comment by auto [ 10/Nov/12 ] |
|
Author: {u'date': u'2012-11-10T22:23:42Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Corrected bson ownership (fixed unit test) |
| Comment by auto [ 10/Nov/12 ] |
|
Author: {u'date': u'2012-11-10T20:55:37Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Included generic bson header (fix compile) |
| Comment by auto [ 09/Nov/12 ] |
|
Author: {u'date': u'2012-11-08T20:33:44Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Introduced type for config.collection docs. |
| Comment by auto [ 06/Nov/12 ] |
|
Author: {u'date': u'2012-10-26T21:49:50Z', u'email': u'alerner@10gen.com', u'name': u'Alberto Lerner'}Message: SERVER-939 Introduced more collection and fields names constants. |
| Comment by Alberto Lerner [ 23/Oct/12 ] |
|
Author: {u'date': u'2012-10-23T12:29:57-07:00', u'name': u'Alberto Lerner', u'email': u'alerner@10gen.com'}Message:Introduced constants to refer to collection names and fields in the config server. |
| Comment by Rakesh Sankar [ 16/Oct/12 ] |
|
This will be a great feature to have and I am eagerly looking forward to it. |
| Comment by Aravind.H.U [ 18/Sep/12 ] |
|
When can I expect this feature , I am badly in need of this , because I create collections dynamically , and collections get filled with data seamlessly . Please let me know when this feature is going to be implemented |
| Comment by Ian Whalen (Inactive) [ 31/Jul/12 ] |
|
Ukolov, this will not be in the 2.2 release. We're still finalizing the feature set for the 2.4 release and will definitely take your concerns and votes here into consideration. |
| Comment by Ukolov Sergey [ 31/Jul/12 ] |
|
Don't you have in plans to implement this in next release? Sorry, but this is very important ticket for me... |
| Comment by Azat Khuzhin [ 25/Apr/12 ] |
|
@eliot how can I do this? (Simple code would be usefull) |
| Comment by Eliot Horowitz (Inactive) [ 25/Apr/12 ] |
|
@azat - you can shard a collection, move the only chunk to where you want it and then disable balancing for that collection. |
| Comment by Azat Khuzhin [ 25/Apr/12 ] |
|
Eliot, how # |
| Comment by Ukolov Sergey [ 24/Apr/12 ] |
|
any news about this? When do you think it will be implemented? With big amount of unsharded collections |
| Comment by Eliot Horowitz (Inactive) [ 05/Jan/12 ] |
|
With |
| Comment by MediaMath Mongo [ 04/Jan/12 ] |
|
|
| Comment by Eliot Horowitz (Inactive) [ 04/Jan/12 ] |
|
|
| Comment by karl seguin [ 19/Aug/11 ] |
|
A particularly useful reason to have this is to be able to use capped collections in a greater number of scenarios. For example, if you want to maintain X logs per server via a capped collection, you need each servers to have its own collection (because you don't want the last 1000 logs in total, you want the last 1000 logs per server.) However, once you do this, you can't shard..and I get sad face. (edit: you can shard, you just need to do it yourself, obviously). |
| Comment by Eliot Horowitz (Inactive) [ 01/Apr/11 ] |
|
No further information that is attached to the ticket, so its currently in 1.9 planning which as the version says is things we're hoping to get in 1.9 |
| Comment by Remon van Vliet [ 01/Apr/11 ] |
|
Any news on ETA and implementation details? This is becoming a problem for us. |
| Comment by Alberto Lerner [ 18/Dec/10 ] |
|
http://www.mongodb.org/display/DOCS/Configuring+Sharding#ConfiguringSharding-ShardingaCollection |
| Comment by Matic [ 18/Dec/10 ] |
|
Thank you for the answers so far. So how can a single collection have documents stored on many shards if it's limited to one shard? |
| Comment by Alberto Lerner [ 17/Dec/10 ] |
|
If you create a new database, the most available shard at that time will be picked to host that entire database. That is , all its collections would be in that shard. If you create another database, possible another shard will be picked for all the collections of that database. With this feature, each of the collections could have a different shard. |
| Comment by Eliot Horowitz (Inactive) [ 17/Dec/10 ] |
|
If you have 1 database, and 1000 collections, all 1000 collections will be on the same shard without this. |
| Comment by Matic [ 17/Dec/10 ] |
|
I don't understand. Could you please explain what scenario is currently unavailable that this feature will make available? |
| Comment by Eliot Horowitz (Inactive) [ 17/Dec/10 ] |
|
No - currently we distribute databases and objects (in a sharded collection), but not collections themselves. |
| Comment by Matic [ 17/Dec/10 ] |
|
What exactly does this feature mean? Doesn't MongoDB already support sharding of collections within a database among nodes in a set? |
| Comment by Eliot Horowitz (Inactive) [ 17/Dec/10 ] |
|
Sorry for continuing to push this, don't want to hack it in a bad way. |