[SERVER-7159] Change DataRecord allocation strategy to quantize within freelist buckets Created: 25/Sep/12 Updated: 27/Oct/15 Resolved: 20/Nov/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | 2.3.1 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Pasette (Inactive) | Assignee: | Ben Becker |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Description |
|
Update allocation strategy to quantize within existing freelist bucket sizes by rounding to closest 1/16th boundary of that bucket range (powers of 2 from 32b to 8MB). For example, if need to allocate 578b, instead allocate 608b (512 + (3 * (1/16 * 512)). Need to test if 1/16 is the proper constant. |
| Comments |
| Comment by auto [ 15/Mar/13 ] |
|
Author: {u'date': u'2013-03-15T10:13:48Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}Message: |
| Comment by auto [ 08/Mar/13 ] |
|
Author: {u'date': u'2013-03-08T16:05:41Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}Message: |
| Comment by auto [ 08/Mar/13 ] |
|
Author: {u'date': u'2013-02-27T23:31:04Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}Message: |
| Comment by auto [ 08/Mar/13 ] |
|
Author: {u'date': u'2013-02-27T23:27:29Z', u'name': u'aaron', u'email': u'aaron@10gen.com'}Message: |
| Comment by auto [ 19/Nov/12 ] |
|
Author: {u'date': u'2012-11-19T19:10:03Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 19/Nov/12 ] |
|
Author: {u'date': u'2012-11-19T19:08:33Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 17/Nov/12 ] |
|
Author: {u'date': u'2012-11-17T02:04:54Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 17/Nov/12 ] |
|
Author: {u'date': u'2012-11-17T00:32:27Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by Ben Becker [ 16/Nov/12 ] |
|
One future change we could look into is modifying the best match logic to prefer the 'lowest' DiskLoc. This may help keep the 'highest' files empty, which could make for a simple way of letting the OS reclaim some file space (with additional work of course). |
| Comment by auto [ 16/Nov/12 ] |
|
Author: {u'date': u'2012-11-16T21:42:31Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 16/Nov/12 ] |
|
Author: {u'date': u'2012-11-16T21:13:47Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 16/Nov/12 ] |
|
Author: {u'date': u'2012-11-16T21:11:27Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by auto [ 16/Nov/12 ] |
|
Author: {u'date': u'2012-11-16T21:05:50Z', u'email': u'ben.becker@10gen.com', u'name': u'Ben Becker'}Message: |
| Comment by Ben Becker [ 15/Nov/12 ] |
|
Thanks, I'm going to move forward with the storage stats as it's going in v2.4, so no need for counters in NSD. |
| Comment by Eric Milkie [ 14/Nov/12 ] |
|
Regarding storing counters, they will need to go in the "reserved" section of NamespaceDetails, and will be an on-disk format change (but one that is easily backwards compatible). Note that you'll need to use the write-intents wrapper when you store the counter, so that journaling works correctly. |
| Comment by Eric Milkie [ 12/Nov/12 ] |
|
On the whole, it sounds reasonable but we'd have to do a full analysis of the performance implications of it. So I would perhaps leave it for later so that we can do smaller incremental changes and more easily see the effects of what we've changed. |
| Comment by Eric Milkie [ 12/Nov/12 ] |
|
I like the idea of increasing the quantization interval for the larger buckets. In particular, the last bucket, which ranges from 4mb to 16mb, needs to have more than 16 quantization boundaries I think. For the first idea, doesn't that mean that you may sometimes get a record that is smaller than requestedSize+padding? I'm having trouble visualizing the math for it. |