[SERVER-5615] Capped collections indexex can grow with no limit. Created: 16/Apr/12  Updated: 22/Mar/16  Resolved: 31/Oct/12

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 1.8.1, 2.0.2, 2.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Tatyana Knaifel Assignee: Gregor Macadam
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 2.6.32-5-amd64 #1 SMP x86_64 GNU/Linux
debian 6.0.1


Attachments: Text File mongo_crash.log    
Issue Links:
Related
related to SERVER-22381 Capped Collection size should account... Closed
Operating System: Linux
Participants:

 Description   

An index , other than "id" on a capped collection can grow with no limitation, and it's space never reused.

I have a capped collection with 7 indexes which was capped to 16G. in several days it's indexes reached 24G in size.

Furthermore, when I ran a validate command on 16G collection (with 24G indexes) - mongod server crashed.

Here is an example on a smaller collection with the same indexes, which I ran with a difference of several minutes:

 db.coll1.stats();
{
	"ns" : "db1.coll1",
	"count" : 2746628,
	"size" : 1029799416,
	"avgObjSize" : 374.93225001711187,
	"storageSize" : 1073745920,
	"numExtents" : 1,
	"nindexes" : 8,
	"lastExtentSize" : 1073745920,
	"paddingFactor" : 1,
	"flags" : 1,
	"totalIndexSize" : 1726918368,
	"indexSizes" : {
		"_id_" : 89142928,
		"date_-1" : 199666096,
		"date_updated_-1" : 192160528,
		"level_1" : 110376000,
		"date_updated_-1_level_1" : 247332176,
		"date_-1_facility_1" : 221119920,
		"date_-1_host_1_facility_1" : 323229984,
		"date_updated_-1_host_1_level_1" : 343890736
	},
	"capped" : 1,
	"max" : 2147483647,
	"ok" : 1
}

several minutes later:

db.coll1.stats();
{
	"ns" : "db1.coll1",
	"count" : 2746972,
	"size" : 1029794112,
	"avgObjSize" : 374.88336684902504,
	"storageSize" : 1073745920,
	"numExtents" : 1,
	"nindexes" : 8,
	"lastExtentSize" : 1073745920,
	"paddingFactor" : 1,
	"flags" : 1,
	"totalIndexSize" : 1728529040,
	"indexSizes" : {
		"_id_" : 89142928,
		"date_-1" : 199935904,
		"date_updated_-1" : 192422160,
		"level_1" : 110343296,
		"date_updated_-1_level_1" : 247479344,
		"date_-1_facility_1" : 221455136,
		"date_-1_host_1_facility_1" : 323614256,
		"date_updated_-1_host_1_level_1" : 344136016
	},
	"capped" : 1,
	"max" : 2147483647,
	"ok" : 1
}

also I was trying to bring the extent information for this collection - but the server crashes again (log of that crash attached)



 Comments   
Comment by Ramon Fernandez Marina [ 22/Mar/16 ]

tmueller@ivugmbh.de, SERVER-22381 is open to consider limiting the size of indexes in capped collections, but it's unclear at the moment if it will be implemented. If this is an issue for you you may want to investigate the use of TTL indexes instead.

Regards,
Ramón.

Comment by Thomas Müller [ 21/Mar/16 ]

Today I had the same issue with a capped collection on a MongoDB instance with version 2.6.9.

But unfortunately I didn't have much to offer on this issue: I've deleted the whole database directory and restored a backup.
It was an a remote machine and there wasn't space to backup the database files.

Comment by Gregor Macadam [ 31/Oct/12 ]

OK - apologies for the long delay - I will close this out as cannot reproduce. If it reoccurs please reopen it.
Thanks
Gregor

Comment by Tatyana Knaifel [ 31/Oct/12 ]

I don't have this problem any more, as now the collection does not
have a lot of indexes and all the setup was changed since.

Comment by Gregor Macadam [ 31/Oct/12 ]

Hi Tatyana
Apologies this issue slipped through the cracks somehow - do you still have the collection that was causing the abort when you tried to validate it? It looks from the backtrack as if you were trying to

db.coll1["$_id_"].validate(true)

Is that correct ? I've tried to reproduce with 1.8.1 and 2.0.4 but so far I am unable to - are you able to reproduce it?
Thanks
Gregor

Comment by Tatyana Knaifel [ 28/May/12 ]

Yes Adam, you can see it in the attached log
i was running validate(

{full:true}

) on a big collection, and the server was crashing

Comment by Adam Comerford [ 24/May/12 ]

Tatyana,

I'm not sure what you mean here - have you seen a crash related to running the stats() command?

Adam

Comment by Tatyana Knaifel [ 16/May/12 ]

Hi

this ticket can be treated only in a regard of server crash when trying to make stats() on a large collection and retrieve of the extent information.

we have removed unnecessary indexes from the collections, and they stopped growing at some point.

Comment by Alvin Richards (Inactive) [ 20/Apr/12 ]

The size of the storage footprint went up as the new capped collection was created. To reduce the disk footprint you will need to run a database repair.

Comment by Tatyana Knaifel [ 19/Apr/12 ]

Hi Eliot,

I have noticed that somehow my original collection stopped to be capped, and while I was trying to make it capped again, I have found several more strange (from my opinion) issues.

So the total size of my DB (before i have started this operation) was 58.1904296875GB
the collection db1.coll_orig was 21G
{
"ns" : "db1.coll_orig",
"count" : 63527918,
"size" : 23373144976,
"avgObjSize" : 367.9192662350433,
"storageSize" : 26221604752,
"numExtents" : 43,
"nindexes" : 3,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 14835785328,
"indexSizes" :

{ "_id_" : 2061153248, "date_updated_-1_level_1" : 5438585264, "date_updated_-1_host_1_level_1" : 7336046816 }

,
"ok" : 1
}

I have ran a command :
db.runCommand(

{"convertToCapped":"coll_orig", size:17179934720}

);

it completed successfully, but all indexes were dropped and my DB size grew to 72.18359375GB
which I can explain why could that happen, but it doesn't seems to be correct behavior.

{
"ns" : "db1.coll_orig",
"count" : 44756686,
"size" : 16463827380,
"avgObjSize" : 367.8517971594233,
"storageSize" : 17179938688,
"numExtents" : 9,
"nindexes" : 0,
"lastExtentSize" : 8523776,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 0,
"indexSizes" : {

},
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}

after I have recreated the indexes - they are now taking 6G and I'll continue monitoring it further.
{
"ns" : "db1.coll_orig",
"count" : 44760473,
"size" : 16463766828,
"avgObjSize" : 367.8193219271834,
"storageSize" : 17179938688,
"numExtents" : 9,
"nindexes" : 3,
"lastExtentSize" : 8523776,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 6684910176,
"indexSizes" :

{ "date_updated_-1_level_1" : 2250501232, "_id_" : 1308037360, "date_updated_-1_host_1_level_1" : 3126371584 }

,
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}

Comment by Tatyana Knaifel [ 18/Apr/12 ]

Hi Eliot,

I'm still observing it.
When I have found so large indexes I have started all the DB from
scratch and on the problematic collection (the 16 G one) have removed
several indexes, so it remains with only 3 indexes.

And created a similar to it collection which is capped for 1G (with 8
indexes)

the small collection is currently 1G data and 1.7G indexes - so it seems
that in this case (there is no reads from this collection) the problem can
not be reproduced.
The index size have remained the same since yesterday:
db.coll1.stats()
{
"ns" : "db1.coll1",
"count" : 2823241,
"size" : 1028573632,
"avgObjSize" : 364.32370881550673,
"storageSize" : 1073745920,
"numExtents" : 1,
"nindexes" : 8,
"lastExtentSize" : 1073745920,
"paddingFactor" : 1,
"flags" : 1,
"totalIndexSize" : 1718333568,
"indexSizes" : {
"id" : 91620256,
"date_-1" : 197622096,
"date_updated_-1" : 194948544,
"level_1" : 106664096,
"date_updated_-1_level_1" : 251027728,
"date_-1_facility_1" : 220727472,
"date_-1_host_1_facility_1" : 308153440,
"date_updated_-1_host_1_level_1" : 347569936
},
"capped" : 1,
"max" : 2147483647,
"ok" : 1
}

the 16G capped collection has currently 17G size and 11G indexes (only 3
indexes out of 8 have remained there)
here are the stats:

db.coll_orig.stats()
{
"ns" : "db1. coll_orig",
"count" : 49730979,
"size" : 18336880248,
"avgObjSize" : 368.7214813929161,
"storageSize" : 19782324160,
"numExtents" : 40,
"nindexes" : 3,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"flags" : 0,
"totalIndexSize" : 11632019728,
"indexSizes" : {
"id" : 1613517248,
"date_updated_-1_level_1" : 4258101680,
"date_updated_-1_host_1_level_1" : 5760400800
},
"ok" : 1
}

All our DB have 10 capped collections of a total of 29G size.

before I have deleted it it was taking 350G space on disk and when i was
running .stats() function on the problematic collection the server was
crashing. all other collections where less than 50G size (together with
their indexes)

it took us about 1 month to get from 29G to 350, so it takes some time.
I'm observing it on a daily basis now.

If I'll not be able to reproduce the problem (using smaller size
collection) within several weeks, I'll add the indexes to the original
collection and will be able to let you connect to our server to check it
out.

Comment by Eliot Horowitz (Inactive) [ 17/Apr/12 ]

@tatyana - in the absolute pedantic case.
Now in your case that's probably for from what's actually going to happen, just trying to explain the theory.

How is the size now?

Comment by Tatyana Knaifel [ 17/Apr/12 ]

Hi Elliot,

So you say that it is possible that for collection of 16 G we will see 256G
indexes ?
if there are 8 indexes on that collection (one of them is the index on _id)

Comment by Eliot Horowitz (Inactive) [ 16/Apr/12 ]

50% is the amount of wasted space.
An index could theoretically be ~ the size of the data.
So I could imagine a pedantic case of a capped collection where total spaces was

N = data size
M = # of indexes

size = N + ( M * N * 2 )

Now, that is the totally pedantic case, but theoretically possible.

Comment by tatyanak@wix.com [ 16/Apr/12 ]

I'll keep that collection for several more days and send you the stats then.
the issue we were observing on a larger collection that it grow (together with the indexes) to 300G instead of 16G .
but from what you say it should not be larger than 500% (8 indexes each 50% + the size of the capped collection itself) from 16G which is 80G.

Comment by Eliot Horowitz (Inactive) [ 16/Apr/12 ]

Indexes don't count within the capped size.
The capped size is for data alone.

Indexes do use space, but b-trees have up to 50% extra space because of balancing.
The guarantee is that it'll never be more than 50%.

Generated at Thu Feb 08 03:09:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.