[SERVER-7733] DB file growing fast Created: 21/Nov/12 Updated: 08/Mar/13 Resolved: 19/Dec/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Blocker - P1 |
| Reporter: | larry.loi | Assignee: | Ian Daniel |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
production environment - Linux prd-mcms-db02 2.6.18-274.el5 x86_64 x86_64 GNU/Linux, 4 CPU 12GB memory; 60GB raid 10 disk * 2 VM server (shard server implementation), 8 app server running mongos connect to them |
||
| Issue Links: |
|
||||||||
| Participants: | |||||||||
| Description |
|
the file size grows fast that doesn't as our expect. data and index size under 20GB but is create over 60GB data files ever I purger old data from database day by day. [root@prd-mcms-db01 ~]# df -k /dev/vx/dsk/prd-mcms-db0102_dg01/mongo01vol /dev/vx/dsk/prd-mcms-db0102_dg02/mongo02vol mongos> show dbs shards: { "_id" : "shard0000", "host" : "prd-mcms-vdb01.prod.laxigames.com:27020" } { "_id" : "shard0001", "host" : "prd-mcms-vdb01.prod.laxigames.com:27021" } { "_id" : "shard0002", "host" : "prd-mcms-vdb02.prod.laxigames.com:27023" } { "_id" : "shard0003", "host" : "prd-mcms-vdb02.prod.laxigames.com:27022" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "fryatt_mcms_production", "partitioned" : true, "primary" : "shard0000" } fryatt_mcms_production.counter_data chunks: |
| Comments |
| Comment by larry.loi [ 23/Dec/12 ] | ||||||||||||||||||||
|
Ian, it is fine for me. please close the case. thanks a lot. | ||||||||||||||||||||
| Comment by Ian Daniel [ 10/Dec/12 ] | ||||||||||||||||||||
|
Hi Larry, Do you have any follow-up questions, or shall I resolve this issue? Kind regards, | ||||||||||||||||||||
| Comment by Ian Daniel [ 04/Dec/12 ] | ||||||||||||||||||||
|
Hi Larry, You can check whether usePowerOf2Sizes is true by running db.<collection>.stats(). For your counter_data collection, the command would be:
This will give you output similar to the following:
Examine the "userFlags" field in the output. If it is set to 1, usePowerOf2Sizes is enabled. If it is set to 0, usePowerOf2Sizes is disabled. You can run this command on the nodes of each shard to assure yourself that the option is enabled on each shard. Kind regards, | ||||||||||||||||||||
| Comment by larry.loi [ 25/Nov/12 ] | ||||||||||||||||||||
|
Eliot, With the below information, How should I know each shard is 100% certain? I had just tested delete and insert operations. , , , }, | ||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 24/Nov/12 ] | ||||||||||||||||||||
|
If you run db.stats() on a collection you can see. | ||||||||||||||||||||
| Comment by larry.loi [ 24/Nov/12 ] | ||||||||||||||||||||
|
How should I check it is enabled or not? how is the below error message? ) }, " | ||||||||||||||||||||
| Comment by larry.loi [ 24/Nov/12 ] | ||||||||||||||||||||
|
one question. should I doing it like that? ) }, | ||||||||||||||||||||
| Comment by larry.loi [ 23/Nov/12 ] | ||||||||||||||||||||
|
Hi Eliot, thanks for you information and we will test it out soon. | ||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 22/Nov/12 ] | ||||||||||||||||||||
|
It is not append only. To avoid this, you can use a new option for allocating consistent sizes: To remove the current fragmentation, you'll probably want to do compact. Power of 2 allocation should prevent further fragmentation. | ||||||||||||||||||||
| Comment by larry.loi [ 22/Nov/12 ] | ||||||||||||||||||||
|
Do you mean new data only append and create new data files? it wouldn't reuse the free spaces even I purged old data? it sounds compact command is rebuild collection only not the whole database. downtime is need also. repairDatabase() command need more physical disk spaces. If database disk partition is over 90% full, it is hard to repair Database. How about the compact command. is it also need additional spaces? | ||||||||||||||||||||
| Comment by Thomas Rueckstiess [ 22/Nov/12 ] | ||||||||||||||||||||
|
Hi Larry, You're saying that you purge old data on a daily basis. This could be the reason why the database is taking more space than the actual data. MongoDB doesn't automatically defragment your database. When you delete old entries, they can leave holes that won't get filled anymore and the size on disk can be much larger than the actual data size. You can try to defragment the collection with the compact command. This command rewrites the whole collection and drops all indexes and re-builds them. It will also free up unused space. Warning: The compact command has severe impact on the performance and should not be done in production, as it blocks the database for the duration of the compact. Only use it in a scheduled maintenance period or by stepping the node down, taking it out of the replica set, compacting, then adding the node back into the set. I hope this answers your question. Let me know if there is anything else we can do to help. Regards, | ||||||||||||||||||||
| Comment by larry.loi [ 21/Nov/12 ] | ||||||||||||||||||||
|
data file could growth show fast like that. data size only around 20GB. and how come it took me over 60GB disk spaces? mongos> show dbs , , , }, | ||||||||||||||||||||
| Comment by larry.loi [ 21/Nov/12 ] | ||||||||||||||||||||
|
from mongos log in application server, it shows the below |