[SERVER-7159] Change DataRecord allocation strategy to quantize within freelist buckets Created: 25/Sep/12  Updated: 27/Oct/15  Resolved: 20/Nov/12

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.2.0
Fix Version/s: 2.3.1

Type: Improvement Priority: Major - P3
Reporter: Daniel Pasette (Inactive) Assignee: Ben Becker
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-3022 Paging in the Extent $freelist chain ... Closed
Related
related to DOCS-1357 Quantize changes need to be reflected... Closed
is related to SERVER-2958 Freelist algorithm causes storage fra... Closed
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Update allocation strategy to quantize within existing freelist bucket sizes by rounding to closest 1/16th boundary of that bucket range (powers of 2 from 32b to 8MB).

For example, if need to allocate 578b, instead allocate 608b (512 + (3 * (1/16 * 512)).

Need to test if 1/16 is the proper constant.



 Comments   
Comment by auto [ 15/Mar/13 ]

Author:

{u'date': u'2013-03-15T10:13:48Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7159 Unit tests for record size quantization and record allocation.
Branch: master
https://github.com/mongodb/mongo/commit/b3e4a86d28b0bec87358e9460183dbb1fab2d159

Comment by auto [ 08/Mar/13 ]

Author:

{u'date': u'2013-03-08T16:05:41Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7159 Enhance auditing support for record size quantization and deleted list.
Branch: master
https://github.com/mongodb/mongo/commit/9791f3f0e8229acb54d763ae0b5788eb41a34d0d

Comment by auto [ 08/Mar/13 ]

Author:

{u'date': u'2013-02-27T23:31:04Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7159 Adjust comments and add minor refactoring for record size quantization code.
Branch: master
https://github.com/mongodb/mongo/commit/c156062309f03e4f5392b877af2df06b288b827f

Comment by auto [ 08/Mar/13 ]

Author:

{u'date': u'2013-02-27T23:27:29Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}

Message: SERVER-7159 Add Random.genNormal() for generating random normal values in tests.
Branch: master
https://github.com/mongodb/mongo/commit/51c0ec740e7dc3eb112ae39e7d248e98e1571622

Comment by auto [ 19/Nov/12 ]

Author:

{u'date': u'2012-11-19T19:10:03Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: use isCapped() accessor method
Branch: master
https://github.com/mongodb/mongo/commit/74da78e550430d742a29439bf7d126ea0ff3e739

Comment by auto [ 19/Nov/12 ]

Author:

{u'date': u'2012-11-19T19:08:33Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: use 1mb boundary for ^2 allocs larger than 4mb
Branch: master
https://github.com/mongodb/mongo/commit/d4c087bcafbd94ae038dc9cd2523bac3f98ebafd

Comment by auto [ 17/Nov/12 ]

Author:

{u'date': u'2012-11-17T02:04:54Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: compensate for increased liklihood of in-place updates due to quantization
Branch: master
https://github.com/mongodb/mongo/commit/e79049861baa58b2e8292eb3c70d6a7859b84336

Comment by auto [ 17/Nov/12 ]

Author:

{u'date': u'2012-11-17T00:32:27Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: don't quantize capped allocations
Branch: master
https://github.com/mongodb/mongo/commit/f8f684a8a3e727ed5baad22433333963e2817346

Comment by Ben Becker [ 16/Nov/12 ]

One future change we could look into is modifying the best match logic to prefer the 'lowest' DiskLoc. This may help keep the 'highest' files empty, which could make for a simple way of letting the OS reclaim some file space (with additional work of course).

Comment by auto [ 16/Nov/12 ]

Author:

{u'date': u'2012-11-16T21:42:31Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159 error codes
Branch: master
https://github.com/mongodb/mongo/commit/9d268547290aed31dcccd4cc9fe4dfaf7effc3e7

Comment by auto [ 16/Nov/12 ]

Author:

{u'date': u'2012-11-16T21:13:47Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: return early if a DeletedRecord matches the requested size
Branch: master
https://github.com/mongodb/mongo/commit/294bce818add5c4bc59f7a7e8f5680bcdecff6e3

Comment by auto [ 16/Nov/12 ]

Author:

{u'date': u'2012-11-16T21:11:27Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: quantize record allocation to nearest 1/16th bucket size
Branch: master
https://github.com/mongodb/mongo/commit/0265277c79d818d09ed54ea815717c7b4d041778

Comment by auto [ 16/Nov/12 ]

Author:

{u'date': u'2012-11-16T21:05:50Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}

Message: SERVER-7159: make largest freelist bucket 16mb
Branch: master
https://github.com/mongodb/mongo/commit/6b5cf42aeebe056b8905ad82b79d0612e7d8d7c3

Comment by Ben Becker [ 15/Nov/12 ]

Thanks, I'm going to move forward with the storage stats as it's going in v2.4, so no need for counters in NSD.

Comment by Eric Milkie [ 14/Nov/12 ]

Regarding storing counters, they will need to go in the "reserved" section of NamespaceDetails, and will be an on-disk format change (but one that is easily backwards compatible). Note that you'll need to use the write-intents wrapper when you store the counter, so that journaling works correctly.

Comment by Eric Milkie [ 12/Nov/12 ]

On the whole, it sounds reasonable but we'd have to do a full analysis of the performance implications of it. So I would perhaps leave it for later so that we can do smaller incremental changes and more easily see the effects of what we've changed.

Comment by Eric Milkie [ 12/Nov/12 ]

I like the idea of increasing the quantization interval for the larger buckets. In particular, the last bucket, which ranges from 4mb to 16mb, needs to have more than 16 quantization boundaries I think.

For the first idea, doesn't that mean that you may sometimes get a record that is smaller than requestedSize+padding? I'm having trouble visualizing the math for it.

Generated at Thu Feb 08 03:13:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.