[SERVER-9489] Recreating a dropped capped collection allocates more physical space Created: 27/Apr/13  Updated: 27/Jan/17  Resolved: 05/Dec/14

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.4.2
Fix Version/s: 2.5.2

Type: Bug Priority: Major - P3
Reporter: Asya Kamsky Assignee: Unassigned
Resolution: Done Votes: 4
Labels: mmapv1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File cappedreuse.js    
Issue Links:
Depends
Related
related to SERVER-6954 FlushViewOfFile with error 487 when t... Closed
is related to DOCS-4097 Oplog resize can cause out of disk space Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

After dropping capped collection of certain size, I would expect recreating it to reuse the space but dropping a 5GB capped collection and then recreating it always allocates two new physical 2GB db.x files.

> db.stats(1024*1024)
{
	"db" : "cap",
	"collections" : 9,
	"objects" : 8732496,
	"avgObjSize" : 203.08234896414496,
	"dataSize" : 1691,
	"storageSize" : 5130,
	"numExtents" : 12,
	"indexes" : 19,
	"indexSize" : 1947,
	"fileSize" : 10299,
	"nsSizeMB" : 16,
	"dataFileVersion" : {
		"major" : 4,
		"minor" : 5
	},
	"ok" : 1
}
> db.sms_message_event.stats(1024*1024)
{
	"ns" : "cap.sms_message_event",
	"count" : 0,
	"size" : 0,
	"storageSize" : 5120,
	"numExtents" : 4,
	"nindexes" : 1,
	"lastExtentSize" : 0,
	"paddingFactor" : 1,
	"systemFlags" : 1,
	"userFlags" : 0,
	"totalIndexSize" : 0,
	"indexSizes" : {
		"_id_" : 0
	},
	"capped" : true,
	"max" : NumberLong("9223372036854775807"),
	"ok" : 1
}
> db.sms_message_event.drop()
true
cap@local(2.4.2-rc0) > db.stats(1024*1024)
{
	"db" : "cap",
	"collections" : 8,
	"objects" : 83,
	"avgObjSize" : 58.795180722891565,
	"dataSize" : 0,
	"storageSize" : 10,
	"numExtents" : 8,
	"indexes" : 13,
	"indexSize" : 0,
	"fileSize" : 12346,
	"nsSizeMB" : 16,
	"dataFileVersion" : {
		"major" : 4,
		"minor" : 5
	},
	"ok" : 1
}
cap@local(2.4.2-rc0) > db.createCollection("sms_message_event",{capped:true, size:5*1024*1024*1024})
{ "ok" : 1 }
cap@local(2.4.2-rc0) > db.stats(1024*1024)
{
	"db" : "cap",
	"collections" : 9,
	"objects" : 86,
	"avgObjSize" : 59.395348837209305,
	"dataSize" : 0,
	"storageSize" : 5130,
	"numExtents" : 12,
	"indexes" : 14,
	"indexSize" : 0,
	"fileSize" : 16440,
	"nsSizeMB" : 16,
	"dataFileVersion" : {
		"major" : 4,
		"minor" : 5
	},
	"ok" : 1
}

Looking in my /data/db after each new drop followed by create I have a new file for that DB:

asyasmacbook:db asya13$ ls -lhu cap*
-rw-------  1 asya13  staff    64M Apr 27 10:16 cap.0
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.1
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.2
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.3
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.4
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.5
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.6
-rw-------  1 asya13  staff    16M Apr 27 10:16 cap.ns
asyasmacbook:db asya13$ ls -lhu cap*
-rw-------  1 asya13  staff    64M Apr 27 10:16 cap.0
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.1
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.2
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.3
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.4
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.5
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.6
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.7
-rw-------  1 asya13  staff    16M Apr 27 10:16 cap.ns
asyasmacbook:db asya13$ ls -lhu cap*
-rw-------  1 asya13  staff    64M Apr 27 10:22 cap.0
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.1
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.2
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.3
-rw-------  1 asya13  staff   2.0G Apr 27 10:09 cap.4
-rw-------  1 asya13  staff   2.0G Apr 27 10:22 cap.5
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.6
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.7
-rw-------  1 asya13  staff   2.0G Apr 27 10:16 cap.8
-rw-------  1 asya13  staff    16M Apr 27 10:22 cap.ns

Here is a repeat showing two more files and times of creation to the second:

asyasmacbook:db asya13$ ls -ltrhUT cap*
-rw-------  1 asya13  staff    16M Apr 27 08:45:19 2013 cap.ns
-rw-------  1 asya13  staff    64M Apr 27 08:45:19 2013 cap.0
-rw-------  1 asya13  staff   2.0G Apr 27 09:02:42 2013 cap.1
-rw-------  1 asya13  staff   2.0G Apr 27 09:02:48 2013 cap.2
-rw-------  1 asya13  staff   2.0G Apr 27 09:02:54 2013 cap.3
-rw-------  1 asya13  staff   2.0G Apr 27 09:39:48 2013 cap.4
-rw-------  1 asya13  staff   2.0G Apr 27 09:39:54 2013 cap.5
-rw-------  1 asya13  staff   2.0G Apr 27 10:08:57 2013 cap.6
-rw-------  1 asya13  staff   2.0G Apr 27 10:16:22 2013 cap.7
-rw-------  1 asya13  staff   2.0G Apr 27 10:16:27 2013 cap.8
-rw-------  1 asya13  staff   2.0G Apr 27 10:25:11 2013 cap.9
-rw-------  1 asya13  staff   2.0G Apr 27 10:25:16 2013 cap.10



 Comments   
Comment by Ramon Fernandez Marina [ 05/Dec/14 ]

This issue no longer reproduces on 2.6.5 or 2.8.0-rc0. Will find the exact release when it was fixed and resolve.

Comment by mongoler [ 18/Jul/13 ]

any one have idea to deal this problem temporary?

Comment by Pawe? Stawicki [ 03/Jun/13 ]

db.coll.stats look OK. Here it is (notice 1024*1024 - values in megabytes):

data-db:PRIMARY> db.sms_message_event.stats(1024*1024)
{
    "ns" : "data.sms_message_event",
    "count" : 8723300,
    "size" : 1737,
    "avgObjSize" : 0.00019912189194456226,
    "storageSize" : 5120,
    "numExtents" : 3,
    "nindexes" : 6,
    "lastExtentSize" : 1026,
    "paddingFactor" : 1,
    "systemFlags" : 1,
    "userFlags" : 0,
    "totalIndexSize" : 2534,
    "indexSizes" : {
        "_id_" : 395,
        "t_1_when_-1" : 475,
        "smsc_message_id_1" : 185,
        "user_id_1_t_1_when_1" : 481,
        "message_id_1" : 318,
        "virtual_number_recipient_when_index" : 678
    },
    "capped" : true,
    "max" : 2147483647,
    "ok" : 1
}

Comment by Scott Hernandez (Inactive) [ 03/Jun/13 ]

Pawel, can you provide db.coll.stats() please?

Comment by adamw [ 30/Apr/13 ]

But why would the indexes grow (indefinitely?), if the documents are removed because of the cap?

Comment by Asya Kamsky [ 30/Apr/13 ]

Pawel: it's possible that the new files are allocated for the growing indexes in your case.

Comment by Pawe? Stawicki [ 29/Apr/13 ]

It's slightly different than the problem I have. Here collection is dropped, while I was inserting documents to mine, so that it hit the cap and should not grow anymore, but it still was creating new files and using more disk space.

I believe it's caused by the same bug, can you confirm?

Comment by Asya Kamsky [ 28/Apr/13 ]

js test that creates a 2GB capped collection, drops it, then creates it again and compares db.stats - 2.2 and 2.4 both increase file size from 4GB after first capped collection is created to 6GB

Generated at Thu Feb 08 03:20:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.