[SERVER-8078] Mongo databases fileSize grows without bound, even though total dataSize doesn't change Created: 04/Jan/13 Updated: 15/Aug/14 Resolved: 01/Apr/13 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 2.2.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | James Blackburn | Assignee: | sam.helman@10gen.com |
| Resolution: | Duplicate | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Operating System: | Linux | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
I have a mongod node which is using much more disk space than stored object space. On Wednesday, I --repair'd the database, which brought the storageSize down to ~dataSize, now storageSize is back to 44x the dataSize! In this case, dataSize is ~480Mb, storageSize is ~21Gb, and there are ~24Gb of database files.
I'll attach some python code that reproduces the problem. |
| Comments |
| Comment by James Blackburn [ 01/Apr/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
powerOf2Size still gives 4x overhead. We're working around by compacting frequently, for the moment - although this is provoking other problems. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 01/Apr/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi James, I'm marking this as a duplicate of | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 13/Feb/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Sam, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by sam.helman@10gen.com [ 07/Feb/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello James, I took a look at | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 31/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks Sam. I understand that our usage can cause fragmentation, the main issue is that the fragmented free space doesn't seem to get reused. Perhaps this is For the moment we've implemented compact when deletedCount is bigger than a threshold, and hoping this resolves the issue long term. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by sam.helman@10gen.com [ 31/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
With regard to $push - even if this operation is done on small arrays, it can cause data fragmentation since it increases the size of the document, often causing it to have to be moved. Using usePowerOfTwoSizes should help mitigate the issue if you are not growing these arrays very much, because it means that the documents will often be padded. However, it will still cause some documents to overrun their allocated space and have to be moved. Combined with deletes, this will inevitably cause fragmentation, which will lead to a larger fileSize/dataSize ratio. The fileSize/dataSize ratio can also be increased by MongoDB's practice of pre-allocating files to hold future data. In your case, it seems that usePowerOfTwoSizes is effectively controlling the situation (since the ratio becomes stable). In the future, as our free-list algorithm and compaction practices improve with new releases, deletion and document movement will cause less fragmentation. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 28/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I just ran it again with usePowerOf2Sizes. It took 10 mins to hit a fileSize of 8519680000, then was stable at that size for 1hr 10mins (at which point I killed it). 8GB still seems big (to me) to store 2 GB of data. However it's not as bad as the 24Gb to store 480Mb we saw in real life.
That's mostly right. We're chunking data (to workaround MongoDB's maximum document size issue), and end up with documents of similar but varying sizes. Once inserted the documents don't grow greatly. However we do periodically delete documents in the collection, so end up fragmenting it / the free space list. Although we do modify the contents of the embedded array with $push / $pull, the size of this set is small, and shouldn't grow quickly. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by sam.helman@10gen.com [ 24/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello, Sorry for the delay in responding. Taking a look at your mongo_hammer.py script, it looks as if your operations do a huge amount of pushing and pulling from embedded arrays, as well as deleting documents. These are situations which cause MongoDB to have a lot of unused space relative to the documents, especially when the documents you are adding and deleting are of somewhat indeterminate size because of the salt you are adding. Please let me know if you think I have analyzed your script incorrectly. If you let the version that has usePowerOf2Sizes turned on run more than once, does the problem get steadily worse or does it even out eventually as the existing buckets are more easily used? (You may have to run it a large number of times so that enough buckets get created). Additionally, does your system require an arbitrary but practically large number of pushes and pulls from embedded documents? Since this type of situation tends to lead to lots of unused space, we often recommend a change in schema to remove the issue. For instance, replacing large embedded arrays with collections is often a good idea, especially since the maximum BSON document size is 16MB and you want to avoid hitting this limit. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 16/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Sam, My main worry is the ratio between fileSize and dataSize. I.e. it seems wrong that the fileSize grows and grows while the amount of data being stored doesn't change.
Smaller numbers are better here. Looking at 1,6 10% of fileSize is data. The rest is just empty wasted space – 14GB to store 1.6GB of data. As this is for 864 objects, on average we're using 18.8MB of space to store each 1.9MB object. It seemed to me that with usePowerOf2Sizes fileSize grew faster. Perhaps this was incorrect. Certainly userPowerOf2Sizes doesn't seem to mitigate the problem. I'm not sure how useful storageSize is, given it's much less than actual space used on disk, and much more than dataSize, it doesn't mean a whole lot to me. Cheers, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by sam.helman@10gen.com [ 16/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
How many runs of mongo_hammer.py were run before this last db.stats() output? It seems to have a better fileSize/storageSize ratio than the first db.stats() from your first comment, but a worse ratio than the last db.stats() from the same comment. Is this the ratio you are concerned about? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 07/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
And the space consumed by the files is increasing at roughly twice the rate of the "storageSize". | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 07/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Adding:
To the top of the script makes the problem worse. Space is being leaked at a faster rate AFAICS using this test. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Randolph Tan [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This is because usePowerOf2Sizes will use fix size extents and it can use up more space than you normally do if you don't have it on, especially if the document sizes are around 2^x + y where y is a very small integer value. This can introduce lots of unused space for append only workload, which would not experience the problem you are having (this also applies to workload types that perform lots of deletes). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks Randolph. So you're saying Mongo leaks disk space without this option? If this is the case then shouldn't it be the default? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
An even simpler repro case, using only one collection in the database. Number of objects is <1000 and dataSize < 2G, disk space used continues to grow. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Randolph Tan [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This has to do with how the data file manager manages space. You can also try using usePowerOf2Sizes (http://docs.mongodb.org/manual/reference/command/collMod/#usePowerOf2Sizes) and see if it helps to mitigate the issues you are seeing. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Also note that although we only insert 2G of data, the starting space usage is 6G. We're less worried about this than the space growth... | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We're attempting to store data of this magnitude, and frequent change rate. Should we compact frequently, or must we plan to repair replicas frequently? | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
In shared memory it grew to 16GB in 21 minutes. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Blackburn [ 04/Jan/13 ] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
1) Startup mongod (if possible point dbpath at a large in memory filesystem so you can see the issue quickly...) At startup mongo_hammer.py creates ~2GB of data. It then deletes some data, before recreating it. The total data should remain at ~2GB at all times. What you see is:
|