[SERVER-2654] Allows sharded capped collections Created: 01/Mar/11  Updated: 06/Dec/22  Resolved: 27/Jan/22

Status: Closed
Project: Core Server
Component/s: Sharding, Storage
Affects Version/s: None
Fix Version/s: features we're not sure of

Type: Improvement Priority: Major - P3
Reporter: Scott Hernandez (Inactive) Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Won't Do Votes: 26
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-3230 Can capped collection auto increase s... Closed
Related
is related to SERVER-211 TTL collections Closed
Assigned Teams:
Sharding
Participants:

 Description   

Allow creating sharded capped collections where no moves are allowed. This will just allow distributing on insert basically; and also for reads/updates (when possible).

This will be esp. useful when TTL capped collections are supported which will allow for keeping session-data.



 Comments   
Comment by Garaudy Etienne [ 27/Jan/22 ]

We recommend users use TTL Indexes instead of capped collections.

Comment by Jose Luis Pedrosa [ 12/Oct/16 ]

HI everyone,

Actually this feature is (from high level view) the functionality provided by Apache kafka. Each topic (collection) is splitted into different partitions (shards), retention period can be configured by size (as mongo) or by time/duration.
Kafka only supports tail operation from any point.. so this could add some extra benefits.

JL

Comment by Jose Luis Pedrosa [ 14/Aug/14 ]

Hi All,

I have really a lot of interest in this feature, I think this is a huge improvement for a lot of uses cases, as message queue. Using a regular collection does not allow you to stream data.
In order to allow some possible implementation, I think the minimum requirement that would do the trick is: guarantee the order of rows only for a given value of the shard key. Why? Imagine shading a capped collection of events, if you shard the collection by customer Id then you could ensure that you'd receive all the events of the same customer in sequence (you care that customer A first registered and then purchased a product, if in the middle you get an event an event out of sequence for customer B is not relevant for business logic, relative order over shard key is important).

I think in order to support scalability, time ranges (or Min max ObjectId) should be always added to the shard key, let me explain:

Initial Chunk distribution :

{ CustomerId A-C, Date X1 -> X2 } =>  shard000

Split required, Chunk distribution after split:

{ CustomerId A->C, Date X1 -> X2 } => shard000
{ CustomerId A->B, Date X2 -> X3 } => shard000
{ CustomerId B->C, Date X2 -> X3 } => shard001

In other words shard key would have time series to support queries. Streaming (tailable cursors) should ignore the shard key distribution, or mongos should inspect the running tailable cursors to determine if a new shard needs to be added to the streams on every split add new shard.

Also I think that a periodic thread should eliminate/modify shards with the minimum dates of the time series of the shard keys available in the capped collection (as the collection rolls).

Jose Luis

Comment by Attila Tozser [ 11/Jun/14 ]

Yes it is true that to disable the balancing on that collection is much better idea. Using TTL indexes does not solve the initial problem what was to produce much better write performance, like a capped collection.

Comment by Asya Kamsky [ 19/May/14 ]

It may be better to disable balancing on this particular collection, rather than disabling the balancer entirely.

Note that as long as your "fake-capped" collections are the same size on each shard, they will stay the same size, and poor insert distribution will just mean that the hotter shards will roll-off old data faster.

It's probably better to use TTL indexes on the sharded collection - that will expire data as it ages, and has the advantage of actually being supported (this request was filed quite a bit earlier than availability of TTL collections).

Comment by Attila Tozser [ 07/May/14 ]

After you create a sharded collection through mongos , if you log into the shards (i tried without replication) it is able to convert the collections (on the shard side) to capped:

{
	"sharded" : true,
	"warning" : "indexes don't all match - ok if ensureIndex is running",
	"ns" : "xxxxx",
	"count" : 189892,
	"numExtents" : 20,
	"size" : 9067920,
	"storageSize" : 33751040,
	"totalIndexSize" : 14831264,
	"indexSizes" : {
		"_id_" : 6205584,
		"id_hash" : 8625680
	},
	"avgObjSize" : 47.753038569292016,
	"nindexes" : 2,
	"nchunks" : 10,
	"shards" : {
		"s1" : {
			"ns" : "xxxxx",
			"count" : 1964,
			"size" : 70704,
			"avgObjSize" : 36,
			"storageSize" : 102400,
			"numExtents" : 1,
			"nindexes" : 1,
			"lastExtentSize" : 102400,
			"paddingFactor" : 1,
			"systemFlags" : 1,
			"userFlags" : 0,
			"totalIndexSize" : 81760,
			"indexSizes" : {
				"_id_" : 81760
			},
			"capped" : true,
			"max" : NumberLong("9223372036854775807"),
			"ok" : 1
		},
		"s2" : {
			"ns" : "xxxxxx",
			"count" : 1964,
			"size" : 70704,
			"avgObjSize" : 36,
			"storageSize" : 102400,
			"numExtents" : 1,
			"nindexes" : 1,
			"lastExtentSize" : 102400,
			"paddingFactor" : 1,
			"systemFlags" : 1,
			"userFlags" : 0,
			"totalIndexSize" : 81760,
			"indexSizes" : {
				"_id_" : 81760
			},
			"capped" : true,
			"max" : NumberLong("9223372036854775807"),
			"ok" : 1
		},
		"s3" : {
			"ns" : "xxxxx",
			"count" : 62182,
			"size" : 2984816,
			"avgObjSize" : 48,
			"storageSize" : 11182080,
			"numExtents" : 6,
			"nindexes" : 2,
			"lastExtentSize" : 8388608,
			"paddingFactor" : 1,
			"systemFlags" : 1,
			"userFlags" : 1,
			"totalIndexSize" : 4905600,
			"indexSizes" : {
				"_id_" : 2019472,
				"id_hash" : 2886128
			},
			"ok" : 1
		},
		"s4" : {
			"ns" : "xxxxx",
			"count" : 61902,
			"size" : 2971376,
			"avgObjSize" : 48,
			"storageSize" : 11182080,
			"numExtents" : 6,
			"nindexes" : 2,
			"lastExtentSize" : 8388608,
			"paddingFactor" : 1,
			"systemFlags" : 1,
			"userFlags" : 1,
			"totalIndexSize" : 4881072,
			"indexSizes" : {
				"_id_" : 2011296,
				"id_hash" : 2869776
			},
			"ok" : 1
		},
		"s5" : {
			"ns" : "xxxxx",
			"count" : 61880,
			"size" : 2970320,
			"avgObjSize" : 48,
			"storageSize" : 11182080,
			"numExtents" : 6,
			"nindexes" : 2,
			"lastExtentSize" : 8388608,
			"paddingFactor" : 1,
			"systemFlags" : 1,
			"userFlags" : 1,
			"totalIndexSize" : 4881072,
			"indexSizes" : {
				"_id_" : 2011296,
				"id_hash" : 2869776
			},
			"ok" : 1
		}
	},
	"ok" : 1
}

(I made only two shards capped)
That will perform the same way as in the description (sharded capped collections where no moves are allowed):

Misbehaviour:

You should switch off balancer (and use hashed index or something that distributes the data well)
when you update something such things may happen:

mongos> db.xxxx.find({ "a" : 300983 })
{ "_id" : ObjectId("536a3d26311f6f7c9f9078ac"), "a" : 300983 }
mongos>
mongos> db.xxxx.update({},{ "_id" : ObjectId("536a3d26311f6f7c9f9078ac")},{$set:{ "a" : "b" }})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
mongos> db.xxxx.find({ "a" : "b" })
mongos> 
mongos> db.xxxx.find({ "a" : 300983 })
mongos> 
mongos> 
mongos> db.xxxx.find({ "_id" : ObjectId("536a3d26311f6f7c9f9078ac") })
{ "_id" : ObjectId("536a3d26311f6f7c9f9078ac") }
mongos> 

When you try to delete something (it is not possible for capped collections):

db.xxxx.remove({ "a" : 300983 })
WriteResult({
	"nRemoved" : 0,
	"writeError" : {
		"code" : 65,
		"errmsg" : "multiple errors for op : cannot remove from a capped collection: xxxxx :: and :: cannot remove from a capped collection:xxxxx"
	}
})

I have not tried but i assume it is able to build this way replicasets for the shards with some secondaries holding capped collections

I assume this way it never will be officially supported, but still able to do.

Best,

Comment by Pieter Willem Jordaan [ 25/May/13 ]

Perhaps an interesting way to address this, although might not be that efficient or even practical, is to:

  • Shard on _id or date field which inherently will force shards to be date oriented.
  • Run shards in a sort of circular buffer approach, where the oldest shard is physically deleted, and becomes the new shard for newer documents based on some segmentation of the date field.
  • Keep chunk size for shard a typical size for the application. By that I mean have the shard keep say a week's data (this can be approximated to a MB value for chunk size)

What do you think?

Comment by James Gosnell [ 10/May/12 ]

@yosef @dwight This is a common problem set(heavy writes, few reads) that many of us would like to find a solution for. An alternative is something like Redis, but obviously it has its limitations as well (inability to shard easily, no multi-key).

Comment by Yosef Dinerstein [ 10/May/12 ]

@Dwight

A bit more details about my scenario.

It is a very typical logging scenario. There are many applications running on many servers writing log messages into mongo. Once in a while, online queries are performed by developers. Those should be made very fast (< 1 second). Once a day aggregation map/reduce is performed to generate report.
So the scenario is WWWWWWWWRWWWWWWWRWWWWWWWR - many writes, a few reads.

Logs that are a few days old aren't necessary and should be discarded. This why capped collection is used.

The time order is important.

Ejection of old data is important.

The amount of servers that running applications grows and hence the amount of logging grows as well. Eventually, one mongo replica set cannot stand the load.

The online queries filter log messages by originating server, application name and time. We create indexes for those fields and compound indexes to support fast queries that involve combinations of those fields.

Map/Reduce jobs aggregate on the fields like message error level, top stack value, application name, originating server.

Comment by Dwight Merriman [ 10/May/12 ]

@Yosef curious about your use case. Would scatter/gather queries suffice? What is the nature of your queries? Insert volume is high? Time order important? Ejection of old data important?

Comment by Yosef Dinerstein [ 10/May/12 ]

Sharded capped collection is very important for my scenario.

I am using capped collection for logging system. There lots of client writing logs. Unfortunately, in absence of sharding, the solution doesn't scale up. Given large enough amount of logging client, the single logging server is not able to stand the load, which makes cappped collection irrelevant.

Comment by Dwight Merriman [ 21/Mar/12 ]

i am thinking there is a variation on the theme which is perhaps not the same thing, but has some analogies, that could work with sharding. cc's have several interesting properties it would be interesting to take that apart a bit and see where the concepts go. But "as is" they basically aren't compatible period i'd say.

Comment by James Gosnell [ 09/Dec/11 ]

Let a datetime field be designated as the capped field, which is then housed in the mongos, similar to index ranging. The same datetime field would probably also have to be part of the shard key to keep from clumping (so cap deletes are even across shards). Then when the cap is exceeded, a remote remove procedure will be sent to the shards where the oldest datetime fields, which would be 1/n the exceeded byte size, where n is the number of shards, will be deleted.

Comment by Dwight Merriman [ 02/Aug/11 ]

if you just want massive scale and don't mind all writes going to one shard for some dt perhaps it isn't so hard

Comment by Eliot Horowitz (Inactive) [ 02/Mar/11 ]

Not sure this ever makes sense since as the migrate semantics, plus roll off will be very strage.

TTL collections will be supported.

Generated at Thu Feb 08 03:00:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.