[SERVER-37812] Compact doesn't free space Created: 30/Oct/18  Updated: 10/May/23  Resolved: 30/Nov/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrey Kostin Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-76830 Compact does not free space Closed
Operating System: ALL
Participants:

 Description   

I'm using MongoDB 4.0.3 with WiredTiger engine in LXD container over ZFS filesystem. 

> db.runCommand({compact: "fs.chunks"})
{ "ok" : 1 }
> db.stats()
{
    "db" : "test",
    "collections" : 3,
    "views" : 0,
    "objects" : 20633515,
    "avgObjSize" : 8189.559693828221,
    "dataSize" : 168979402786,
    "storageSize" : 458719870976,
    "numExtents" : 0,
    "indexes" : 8,
    "indexSize" : 3488063488,
    "fsUsedSize" : 464063299584,
    "fsTotalSize" : 578740813824,
    "ok" : 1
}
> db.runCommand({compact: "fs.chunks"})
{ "ok" : 1 }
> db.stats()
{
    "db" : "test",
    "collections" : 3,
    "views" : 0,
    "objects" : 20633519,
    "avgObjSize" : 8189.558252617985,
    "dataSize" : 168979405807,
    "storageSize" : 458719866880,
    "numExtents" : 0,
    "indexes" : 8,
    "indexSize" : 3488063488,
    "fsUsedSize" : 464064217088,
    "fsTotalSize" : 578740813824,
    "ok" : 1
}

As you can see, DB didn't release unneeded space.



 Comments   
Comment by Danny Hatcher (Inactive) [ 30/Nov/18 ]

Hello Andrey,

As we've heard nothing back from you, I will now close this ticket.

Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 14/Nov/18 ]

Hello Andrey,

Can you please upload the mongod logs and the diagnostic.data folders from all three nodes to our Uploader via the following link?

https://10gen-httpsupload.s3.amazonaws.com/upload_forms/e90ca8ad-1026-44a8-a833-4db316736a54.html

Thank you,

Danny

Comment by Andrey Kostin [ 08/Nov/18 ]

This collection was created a long time ago. I'm not sure about version I used at that moment but it was one of 3.2 - 3.6.
For all operations I use php script which mostly adds new images to gridfs. About a month ago I decided to remove a big part of this images and to compact collection after that.

I have:
a) master node with ubuntu 16.04, 2x2TB raid with ext4
b) replica node with ubuntu 16.04, 1x2TB disk with ext4 (on lvm volume)
c) desktop with ubuntu 16.04, LXD container on zfs

Collection was successfully compacted on replica node. Then I tried 3 times to do the same with desktop and no success. After that I put master node in secondary state with stepDown and started compact operation which also didn't succeed.

Right now I can only reproduce this issue using master node since at desktop I reduced the size of the storage using mongodump/mongorestore.

Comment by Danny Hatcher (Inactive) [ 07/Nov/18 ]

Hello Andrey,

You mention that this is on MongoDB 4.0.3. Was this collection created while you were on 4.0.3 or was the collection created while you were on a different version (and what version would that be)?

Do you have any way to reproduce this issue in a consistent manner? Could you please explain in more detail as to how you are using GridFS to insert, edit, and delete your data?

Thank you,

Danny

Comment by Andrey Kostin [ 07/Nov/18 ]

{
    "ns" : "test.fs.chunks",
    "size" : 166583740073,
    "count" : 8948623,
    "avgObjSize" : 18615,
    "storageSize" : 456644669440,
    "capped" : false,
    "wiredTiger" : {
        "metadata" : {
            "formatVersion" : 1
        },
        "creationString" : "access_pattern_hint=none,allocation_size=4KB,app_metadata=(formatVersion=1),assert=(commit_timestamp=none,read_timestamp=none),block_allocation=best,block_compressor=snappy,cache_resident=false,checksum=on,colgroups=,collator=,columns=,dictionary=0,encryption=(keyid=,name=),exclusive=false,extractor=,format=btree,huffman_key=,huffman_value=,ignore_in_memory_cache_size=false,immutable=false,internal_item_max=0,internal_key_max=0,internal_key_truncate=true,internal_page_max=4KB,key_format=q,key_gap=10,leaf_item_max=0,leaf_key_max=0,leaf_page_max=32KB,leaf_value_max=64MB,log=(enabled=false),lsm=(auto_throttle=true,bloom=true,bloom_bit_count=16,bloom_config=,bloom_hash_count=8,bloom_oldest=false,chunk_count_limit=0,chunk_max=5GB,chunk_size=10MB,merge_custom=(prefix=,start_generation=0,suffix=),merge_max=15,merge_min=0),memory_page_image_max=0,memory_page_max=10m,os_cache_dirty_max=0,os_cache_max=0,prefix_compression=false,prefix_compression_min=4,source=,split_deepen_min_child=0,split_deepen_per_child=0,split_pct=90,type=file,value_format=u",
        "type" : "file",
        "uri" : "statistics:table:test/collection/18-8782054347731098863",
        "LSM" : {
            "bloom filter false positives" : 0,
            "bloom filter hits" : 0,
            "bloom filter misses" : 0,
            "bloom filter pages evicted from cache" : 0,
            "bloom filter pages read into cache" : 0,
            "bloom filters in the LSM tree" : 0,
            "chunks in the LSM tree" : 0,
            "highest merge generation in the LSM tree" : 0,
            "queries that could have benefited from a Bloom filter that did not exist" : 0,
            "sleep for LSM checkpoint throttle" : 0,
            "sleep for LSM merge throttle" : 0,
            "total size of bloom filters" : 0
        },
        "block-manager" : {
            "allocations requiring file extension" : 0,
            "blocks allocated" : 495664,
            "blocks freed" : 10521,
            "checkpoint size" : 176018661376,
            "file allocation unit size" : 4096,
            "file bytes available for reuse" : 280620048384,
            "file magic number" : 120897,
            "file major version number" : 1,
            "file size in bytes" : 456644669440,
            "minor version number" : 0
        },
        "btree" : {
            "btree checkpoint generation" : 10732,
            "column-store fixed-size leaf pages" : 0,
            "column-store internal pages" : 0,
            "column-store variable-size RLE encoded values" : 0,
            "column-store variable-size deleted values" : 0,
            "column-store variable-size leaf pages" : 0,
            "fixed-record size" : 0,
            "maximum internal page key size" : 368,
            "maximum internal page size" : 4096,
            "maximum leaf page key size" : 2867,
            "maximum leaf page size" : 32768,
            "maximum leaf page value size" : 67108864,
            "maximum tree depth" : 6,
            "number of key/value pairs" : 0,
            "overflow pages" : 0,
            "pages rewritten by compaction" : 0,
            "row-store internal pages" : 0,
            "row-store leaf pages" : 0
        },
        "cache" : {
            "bytes currently in the cache" : 1761798132,
            "bytes read into cache" : 8439450185,
            "bytes written from cache" : 12458370785,
            "checkpoint blocked page eviction" : 0,
            "data source pages selected for eviction unable to be evicted" : 1243,
            "eviction walk passes of a file" : 27159,
            "eviction walk target pages histogram - 0-9" : 1827,
            "eviction walk target pages histogram - 10-31" : 5314,
            "eviction walk target pages histogram - 128 and higher" : 0,
            "eviction walk target pages histogram - 32-63" : 5006,
            "eviction walk target pages histogram - 64-128" : 15012,
            "eviction walks abandoned" : 5332,
            "eviction walks gave up because they restarted their walk twice" : 4,
            "eviction walks gave up because they saw too many pages and found no candidates" : 2885,
            "eviction walks gave up because they saw too many pages and found too few candidates" : 3806,
            "eviction walks reached end of tree" : 9638,
            "eviction walks started from root of tree" : 12028,
            "eviction walks started from saved location in tree" : 15131,
            "hazard pointer blocked page eviction" : 52,
            "in-memory page passed criteria to be split" : 2693,
            "in-memory page splits" : 1348,
            "internal pages evicted" : 88724,
            "internal pages split during eviction" : 36,
            "leaf pages split during eviction" : 9789,
            "modified pages evicted" : 865009,
            "overflow pages read into cache" : 0,
            "page split during eviction deepened the tree" : 0,
            "page written requiring cache overflow records" : 0,
            "pages read into cache" : 501977,
            "pages read into cache after truncate" : 0,
            "pages read into cache after truncate in prepare state" : 0,
            "pages read into cache requiring cache overflow entries" : 0,
            "pages requested from the cache" : 5573782,
            "pages seen by eviction walk" : 18082896,
            "pages written from cache" : 491448,
            "pages written requiring in-memory restoration" : 846315,
            "tracked dirty bytes in the cache" : 0,
            "unmodified pages evicted" : 510177
        },
        "cache_walk" : {
            "Average difference between current eviction generation when the page was last considered" : 0,
            "Average on-disk page image size seen" : 0,
            "Average time in cache for pages that have been visited by the eviction server" : 0,
            "Average time in cache for pages that have not been visited by the eviction server" : 0,
            "Clean pages currently in cache" : 0,
            "Current eviction generation" : 0,
            "Dirty pages currently in cache" : 0,
            "Entries in the root page" : 0,
            "Internal pages currently in cache" : 0,
            "Leaf pages currently in cache" : 0,
            "Maximum difference between current eviction generation when the page was last considered" : 0,
            "Maximum page size seen" : 0,
            "Minimum on-disk page image size seen" : 0,
            "Number of pages never visited by eviction server" : 0,
            "On-disk page image sizes smaller than a single allocation unit" : 0,
            "Pages created in memory and never written" : 0,
            "Pages currently queued for eviction" : 0,
            "Pages that could not be queued for eviction" : 0,
            "Refs skipped during cache traversal" : 0,
            "Size of the root page" : 0,
            "Total number of pages currently in cache" : 0
        },
        "compression" : {
            "compressed pages read" : 85688,
            "compressed pages written" : 77410,
            "page written failed to compress" : 398755,
            "page written was too small to compress" : 15277,
            "raw compression call failed, additional data available" : 0,
            "raw compression call failed, no additional data available" : 0,
            "raw compression call succeeded" : 0
        },
        "cursor" : {
            "bulk-loaded cursor-insert calls" : 0,
            "create calls" : 53,
            "cursor operation restarted" : 10510,
            "cursor-insert key and value bytes inserted" : 12346216877,
            "cursor-remove key bytes removed" : 0,
            "cursor-update value bytes updated" : 0,
            "cursors cached on close" : 0,
            "cursors reused from cache" : 1384799,
            "insert calls" : 874501,
            "modify calls" : 0,
            "next calls" : 0,
            "prev calls" : 1,
            "remove calls" : 0,
            "reserve calls" : 0,
            "reset calls" : 2771903,
            "search calls" : 506416,
            "search near calls" : 0,
            "truncate calls" : 0,
            "update calls" : 0
        },
        "reconciliation" : {
            "dictionary matches" : 0,
            "fast-path pages deleted" : 0,
            "internal page key bytes discarded using suffix compression" : 898185,
            "internal page multi-block writes" : 3754,
            "internal-page overflow keys" : 0,
            "leaf page key bytes discarded using prefix compression" : 0,
            "leaf page multi-block writes" : 11070,
            "leaf-page overflow keys" : 0,
            "maximum blocks required for a page" : 1,
            "overflow values written" : 0,
            "page checksum matches" : 235278,
            "page reconciliation calls" : 879388,
            "page reconciliation calls for eviction" : 856905,
            "pages deleted" : 4152
        },
        "session" : {
            "cached cursor count" : 1750,
            "object compaction" : 0,
            "open cursor count" : 0
        },
        "transaction" : {
            "update conflicts" : 0
        }
    },
    "nindexes" : 3,
    "totalIndexSize" : 452579328,
    "indexSizes" : {
        "files_id_1_n_1" : 246980608,
        "export_1" : 69632,
        "_id_" : 205529088
    },
    "ok" : 1,
    "operationTime" : Timestamp(1541608216, 10),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1541608216, 10),
        "signature" : {
            "hash" : BinData(0,"jBkey/UkqPISsEqBVW0GEI//nFw="),
            "keyId" : NumberLong("6618182060651577345")
        }
    }
}

I have 1059GB of free space available for mongod instance.

Comment by Danny Hatcher (Inactive) [ 06/Nov/18 ]

Hello Andrey,

Because you are using GridFS to store blobs of data, it is very possible that WiredTiger needs to reserve that much space on disk. When you run db.fs.chunks.stats(), what is the value under the heading wiredTiger.block-manager.file bytes available for reuse? If the number there is 0 or very low, then compact will most likely not have any effect on reducing the storage space on your system. Please provide the output of the full db.fs.chunks.stats() command so that we can see if anything else stands out.

Please note that unless you have recently removed a large amount of data from your system, it is not likely that you will see a large reduction of storage used by running the compact command.

Thank you,

Danny

Comment by Andrey Kostin [ 31/Oct/18 ]

> db.fs.chunks.stats().size
152247530612
> db.fs.chunks.stats().storageSize
454824726528

Comment by Andrey Kostin [ 30/Oct/18 ]

If you take a look at db.stats() results you'll see that I have 169GB of data in 458GB storage. There are only 2 collections in this DB, fs.chunks (~166GB) and fs.files (~3GB).

Comment by Ramon Fernandez Marina [ 30/Oct/18 ]

lisio, this is not unexpected behavior, depending on the circumstances; the documentation says:

On WiredTiger, compact attempts to reduce the required storage space for data and indexes in a collection, releasing unneeded disk space to the operating system. The effectiveness of this operation is workload dependent and no disk space may be recovered. This command is useful if you have removed a large amount of data from the collection, and do not plan to replace it.

If you didn't remove large amounts of data from fs.chunks the compact will not reclaim any unused disk space because there may not be any no reclaim.

If there's a way you can reproduce this issue reliably and you believe there's a bug, will you please share the reproducer so we can investigate?

Thanks,
Ramón.

Generated at Thu Feb 08 04:47:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.