[SERVER-13331] GridFS chunks collection should lower chunk size to 255k Created: 24/Mar/14  Updated: 11/Jul/16  Resolved: 24/Mar/14

Status: Closed
Project: Core Server
Component/s: GridFS
Affects Version/s: None
Fix Version/s: 2.4.10, 2.6.0-rc3

Type: Improvement Priority: Critical - P2
Reporter: Tyler Brock Assignee: Tyler Brock
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CXX-149 Lower chunk size to 255k on GridFS im... Closed
Related
related to DOCS-2991 Update GridFS chunk size default to 255k Closed
related to SERVER-12088 Enable usePowerOf2Sizes as server-wid... Closed
is related to SERVER-15856 GridFS chunk size limit is 255KB not ... Closed
Participants:

 Description   
Issue Status as of March 31, 2014

ISSUE SUMMARY

From major release version 2.6 on, the usePowerOf2Sizes option is enabled by default, rounding up the space allocated for a record to the next power of two. This makes the current default for chunk sizes in GridFS, which is 256 KB, a bad choice. The overhead of _id and foreign key for chunk documents increases the size to just over 256 KB and would therefore cause 512 KB space allocation, with almost half of the space wasted.

USER IMPACT

In the 2.4 release cycle, usePowerOf2Sizes is not enabled by default. This only affects users who have manually enabled usePowerOf2Sizes on their GridFS chunks collection.

SOLUTION

The fix is to reduce the default chunk size of GridFS documents to 255 KB. This leaves enough space for the extra fields to still only allocate 256 KB of space for each document.

WORKAROUNDS

Driver versions designed to be used with the 2.6 release will include this fix client-side.
Alternatively, disabling usePowerOf2Sizes also prevents the space overhead but can affect space re-use efficiency, especially in situations were documents are frequently deleted and recreated. This can lead to extent fragmentation.

AFFECTED VERSIONS

All recent production release versions up to 2.4.9 are affected.

PATCHES

The fix is included in the 2.4.10 production release and the 2.6.0-rc3 release candidate, which will evolve into the 2.6.0 production release.

Original Description

Now that the server uses power of 2 by default, if the default chunk size for gridfs is 256k we will almost always be throwing away some storage space. This is because if the bindata field of a chunk will occupy 256k (an exact power of 2), then the _id and foreign key reference to the files collection, etc will take up additional space that will cause the document's allocated storage to be rounded up to 512k (the next power of 2). This would be a huge waste considering it would round up every chunk for a given file.

Instead, if we make the default chunk size 255k then we have an extra 1k to store the _id and other metadata so that when the document is saved we round up to 256k and not 512k upon persisting the document.



 Comments   
Comment by Githook User [ 24/Mar/14 ]

Author:

{u'username': u'TylerBrock', u'name': u'Tyler Brock', u'email': u'tyler.brock@gmail.com'}

Message: SERVER-13331 lower GridFS default chunk size to 255k
(cherry picked from commit 632ae4e7ad70426d5e2d36bd1fee4491eeac16b7)
Branch: v2.4
https://github.com/mongodb/mongo/commit/0080fdc3fea4be834b14eb65d8a334b1d83a00ab

Comment by Githook User [ 24/Mar/14 ]

Author:

{u'username': u'TylerBrock', u'name': u'Tyler Brock', u'email': u'tyler.brock@gmail.com'}

Message: SERVER-13331 lower GridFS default chunk size to 255k

(cherry picked from commit 632ae4e7ad70426d5e2d36bd1fee4491eeac16b7)
Branch: v2.6
https://github.com/mongodb/mongo/commit/258d46ecee93b79a81e5d6d506ded223c69aa30d

Comment by Githook User [ 24/Mar/14 ]

Author:

{u'username': u'TylerBrock', u'name': u'Tyler Brock', u'email': u'tyler.brock@gmail.com'}

Message: SERVER-13331 lower GridFS default chunk size to 255k
Branch: master
https://github.com/mongodb/mongo/commit/632ae4e7ad70426d5e2d36bd1fee4491eeac16b7

Generated at Thu Feb 08 03:31:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.