[SERVER-7733] DB file growing fast Created: 21/Nov/12  Updated: 08/Mar/13  Resolved: 19/Dec/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.2.0
Fix Version/s: None

Type: Question Priority: Blocker - P1
Reporter: larry.loi Assignee: Ian Daniel
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

production environment - Linux prd-mcms-db02 2.6.18-274.el5 x86_64 x86_64 GNU/Linux, 4 CPU 12GB memory; 60GB raid 10 disk * 2 VM server (shard server implementation), 8 app server running mongos connect to them


Issue Links:
Related
is related to DOCS-842 Document what collection.stats() user... Closed
Participants:

 Description   

the file size grows fast that doesn't as our expect. data and index size under 20GB but is create over 60GB data files ever I purger old data from database day by day.
and also found 2 shard server are unavailable. right now.

[root@prd-mcms-db01 ~]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda8 988088 602524 334560 65% /

/dev/vx/dsk/prd-mcms-db0102_dg01/mongo01vol
62809702 29998500 30760744 50% /laxino/mongodb-vdb01

/dev/vx/dsk/prd-mcms-db0102_dg02/mongo02vol
62809702 47496155 14356697 77% /laxino/mongodb-vdb02

mongos> show dbs
admin (empty)
config 0.1875GB
fryatt_logms_production 0.203125GB
fryatt_mcms_production 67.783203125GB
test 0.203125GB
mongos> sh.status()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard0000", "host" : "prd-mcms-vdb01.prod.laxigames.com:27020" } { "_id" : "shard0001", "host" : "prd-mcms-vdb01.prod.laxigames.com:27021" } { "_id" : "shard0002", "host" : "prd-mcms-vdb02.prod.laxigames.com:27023" } { "_id" : "shard0003", "host" : "prd-mcms-vdb02.prod.laxigames.com:27022" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "fryatt_mcms_production", "partitioned" : true, "primary" : "shard0000" }

fryatt_mcms_production.counter_data chunks:
shard0001 136
shard0003 133
shard0000 134
shard0002 211
too many chunks to print, use verbose if you want to force print

{ "_id" : "test", "partitioned" : false, "primary" : "shard0001" } { "_id" : "admin --username admin --password XXXXXXXX", "partitioned" : false, "primary" : "shard0000" } { "_id" : "fryatt_logms_production", "partitioned" : false, "primary" : "shard0003" }

 Comments   
Comment by larry.loi [ 23/Dec/12 ]

Ian,

it is fine for me. please close the case. thanks a lot.

Comment by Ian Daniel [ 10/Dec/12 ]

Hi Larry,

Do you have any follow-up questions, or shall I resolve this issue?

Kind regards,
Ian

Comment by Ian Daniel [ 04/Dec/12 ]

Hi Larry,

You can check whether usePowerOf2Sizes is true by running db.<collection>.stats().

For your counter_data collection, the command would be:

db.counter_data.stats()

This will give you output similar to the following:

> db.account.stats()
{
	"ns" : "test.account",
	"count" : 7,
	"size" : 392,
	"avgObjSize" : 56,
	"storageSize" : 4096,
	"numExtents" : 1,
	"nindexes" : 1,
	"lastExtentSize" : 4096,
	"paddingFactor" : 1,
	"systemFlags" : 1,
	"userFlags" : 1,
	"totalIndexSize" : 8176,
	"indexSizes" : {
		"_id_" : 8176
	},
	"ok" : 1
}

Examine the "userFlags" field in the output. If it is set to 1, usePowerOf2Sizes is enabled. If it is set to 0, usePowerOf2Sizes is disabled.

You can run this command on the nodes of each shard to assure yourself that the option is enabled on each shard.

Kind regards,
Ian

Comment by larry.loi [ 25/Nov/12 ]

Eliot,

With the below information, How should I know each shard is 100% certain? I had just tested delete and insert operations.
mongos> db.stats()
{
"raw" : {
"prd-mcms-vdb01.prod.laxigames.com:27020" :

{ "db" : "fryatt_mcms_production", "collections" : 6, "objects" : 27, "avgObjSize" : 97.62962962962963, "dataSize" : 2636, "storageSize" : 58482688, "numExtents" : 14, "indexes" : 10, "indexSize" : 81760, "fileSize" : 469762048, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb01.prod.laxigames.com:27021" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 100009, "avgObjSize" : 1043.057284844364, "dataSize" : 104315116, "storageSize" : 123949056, "numExtents" : 13, "indexes" : 3, "indexSize" : 29474480, "fileSize" : 469762048, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27022" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 100007, "avgObjSize" : 1037.7284390092693, "dataSize" : 103780108, "storageSize" : 174747648, "numExtents" : 14, "indexes" : 3, "indexSize" : 30610944, "fileSize" : 1006632960, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27023" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 9, "avgObjSize" : 140, "dataSize" : 1260, "storageSize" : 58454016, "numExtents" : 11, "indexes" : 3, "indexSize" : 24528, "fileSize" : 469762048, "nsSizeMB" : 16, "ok" : 1 }

},
"objects" : 200052,
"avgObjSize" : 1040.2251414632196,
"dataSize" : 208099120,
"storageSize" : 415633408,
"numExtents" : 52,
"indexes" : 19,
"indexSize" : 60191712,
"fileSize" : 2415919104,
"ok" : 1
}

Comment by Eliot Horowitz (Inactive) [ 24/Nov/12 ]

If you run db.stats() on a collection you can see.
You can check on each shard to be 100% certain.
Looks like you weren't authenticated in the last attempt.

Comment by larry.loi [ 24/Nov/12 ]

How should I check it is enabled or not? how is the below error message?
mongos> db.runCommand(

{"collMod" : "counter_data" , "usePowerOf2Sizes" : "true" }

)
{
"raw" : {
"prd-mcms-vdb01.prod.laxigames.com:27020" :

{ "usePowerOf2Sizes_old" : true, "errmsg" : "unknown command: $auth", "ok" : 0 }

},
"ok" : 0,
"errmsg" : "

{ prd-mcms-vdb01.prod.laxigames.com:27020: \"unknown command: $auth\" }

"
}

Comment by larry.loi [ 24/Nov/12 ]

one question. should I doing it like that?
mongos> db.runCommand(

{"collMod" : "counter_data" , "usePowerOf2Sizes" : "true" }

)
{
"raw" : {
"prd-mcms-vdb01.prod.laxigames.com:27021" :

{ "usePowerOf2Sizes_old" : false, "ok" : 1 }

},
"ok" : 1
}

Comment by larry.loi [ 23/Nov/12 ]

Hi Eliot, thanks for you information and we will test it out soon.

Comment by Eliot Horowitz (Inactive) [ 22/Nov/12 ]

It is not append only.
When you delete something, the entry goes onto a free list.
However, if you do this a lot with documents of varied size, things can get fragmented.

To avoid this, you can use a new option for allocating consistent sizes:
http://docs.mongodb.org/manual/reference/command/collMod/#usePowerOf2Sizes

To remove the current fragmentation, you'll probably want to do compact.
You can do on secondaries and then rotate around for the current primary.

Power of 2 allocation should prevent further fragmentation.

Comment by larry.loi [ 22/Nov/12 ]

Do you mean new data only append and create new data files? it wouldn't reuse the free spaces even I purged old data? it sounds compact command is rebuild collection only not the whole database. downtime is need also. repairDatabase() command need more physical disk spaces. If database disk partition is over 90% full, it is hard to repair Database. How about the compact command. is it also need additional spaces?

Comment by Thomas Rueckstiess [ 22/Nov/12 ]

Hi Larry,

You're saying that you purge old data on a daily basis. This could be the reason why the database is taking more space than the actual data. MongoDB doesn't automatically defragment your database. When you delete old entries, they can leave holes that won't get filled anymore and the size on disk can be much larger than the actual data size.

You can try to defragment the collection with the compact command. This command rewrites the whole collection and drops all indexes and re-builds them. It will also free up unused space.

Warning: The compact command has severe impact on the performance and should not be done in production, as it blocks the database for the duration of the compact. Only use it in a scheduled maintenance period or by stepping the node down, taking it out of the replica set, compacting, then adding the node back into the set.

I hope this answers your question. Let me know if there is anything else we can do to help.

Regards,
Thomas

Comment by larry.loi [ 21/Nov/12 ]

data file could growth show fast like that. data size only around 20GB. and how come it took me over 60GB disk spaces?

mongos> show dbs
admin (empty)
config 0.1875GB
fryatt_logms_production 0.203125GB
fryatt_mcms_production 67.783203125GB
test 0.203125GB
mongos> db.stats()
{
"raw" : {
"prd-mcms-vdb01.prod.laxigames.com:27020" :

{ "db" : "fryatt_mcms_production", "collections" : 7, "objects" : 189590, "avgObjSize" : 240.01824990769555, "dataSize" : 45505060, "storageSize" : 5157539840, "numExtents" : 41, "indexes" : 11, "indexSize" : 46169872, "fileSize" : 14958985216, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb01.prod.laxigames.com:27021" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 540933, "avgObjSize" : 241.3861901566368, "dataSize" : 130573756, "storageSize" : 4287758336, "numExtents" : 32, "indexes" : 3, "indexSize" : 132132336, "fileSize" : 12812550144, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27023" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 40709929, "avgObjSize" : 240.87430169676787, "dataSize" : 9805975720, "storageSize" : 13375680464, "numExtents" : 37, "indexes" : 3, "indexSize" : 11470739952, "fileSize" : 34276900864, "nsSizeMB" : 16, "ok" : 1 }

,
"prd-mcms-vdb02.prod.laxigames.com:27022" :

{ "db" : "fryatt_mcms_production", "collections" : 3, "objects" : 223951, "avgObjSize" : 241.34477631267555, "dataSize" : 54049404, "storageSize" : 3828215808, "numExtents" : 32, "indexes" : 3, "indexSize" : 56062832, "fileSize" : 10666115072, "nsSizeMB" : 16, "ok" : 1 }

},
"objects" : 41664403,
"avgObjSize" : 240.87958106588016,
"dataSize" : 10036103940,
"storageSize" : 26649194448,
"numExtents" : 142,
"indexes" : 20,
"indexSize" : 11705104992,
"fileSize" : 72714551296,
"ok" : 1
}
mongos>

Comment by larry.loi [ 21/Nov/12 ]

from mongos log in application server, it shows the below
currently at id 000000000000000000000000
Wed Nov 21 16:44:51 [Balancer] distributed lock 'balancer/prd-hgsms-app01.prod.laxigames.com:27007:1353151353:1804289383' acquired, ts : 50ac94832de32f65b0a252d5
Wed Nov 21 16:44:51 [Balancer] shard0001 is unavailable
Wed Nov 21 16:44:51 [Balancer] shard0002 is unavailable
Wed Nov 21 16:44:51 [Balancer] distributed lock 'balancer/prd-hgsms-app01.prod.laxigames.com:27007:1353151353:1804289383' unlocked.
Wed Nov 21 16:44:52 [mongosMain] connection accepted from 127.0.0.1:48362 #100540 (85 connections now open)
Wed Nov 21 16:44:52 [conn100540] end connection 127.0.0.1:48362 (84 connections now open)
Wed Nov 21 16:44:53 [conn98246] DBException in process: socket exception [SEND_ERROR] for 12

Generated at Thu Feb 08 03:15:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.