Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-2958

Freelist algorithm causes storage fragmentation

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL

      Description

      The algorithm used for the freelist groups related sizes into "buckets" that are searched for a free entry. The algorithm stops at 30 entries and then goes to the next bucket. If all buckets are searched then a new extent is allocated. In a high insert / delete environment where the inserts occur throughout the delay and peak and then deletes peak at a separate time (for example a session cache for a web site) this algorithm results in very large freelists where the smallest items filter to the top of each bucket. The freelist becomes filled with items that are never reused and blocking items that can be reused. One option is to allocate only on the bucket size (256, 512, 1024, etc.) which would guarantee that all items in the freelist are reusable. The following pull request https://github.com/mongodb/mongo/pull/37 illustrates how this could be fixed.

        Issue Links

          Activity

          Hide
          dwight_10gen Dwight Merriman added a comment -

          fwiw in 2.2 the compact command has some options for specifying padding that may or may not be helpful for you

          Show
          dwight_10gen Dwight Merriman added a comment - fwiw in 2.2 the compact command has some options for specifying padding that may or may not be helpful for you
          Hide
          rem René M. Andersen added a comment -

          Any news on when we can expect this to be fixed? This causes problems for us because we have a collection with many inserts and deletes occuring all the time. In production, the database size on disk grows with 1GB a week even though the size of the data stays approx. the same.

          Show
          rem René M. Andersen added a comment - Any news on when we can expect this to be fixed? This causes problems for us because we have a collection with many inserts and deletes occuring all the time. In production, the database size on disk grows with 1GB a week even though the size of the data stays approx. the same.
          Hide
          eliot Eliot Horowitz added a comment -

          @rene - you should try usePowerOf2Sizes, should do a lot better http://docs.mongodb.org/manual/reference/command/collMod/

          Show
          eliot Eliot Horowitz added a comment - @rene - you should try usePowerOf2Sizes, should do a lot better http://docs.mongodb.org/manual/reference/command/collMod/
          Hide
          rem René M. Andersen added a comment -

          @Eliot - Ok I will try that. We have a different scenario related to this jira SERVER-2838 where we drop collections (a "fast delete" of aged data) and create new ones for new data. Will the "usePowerOf2Sizes" also make the reusing of the disk space more efficient for these collections?

          Show
          rem René M. Andersen added a comment - @Eliot - Ok I will try that. We have a different scenario related to this jira SERVER-2838 where we drop collections (a "fast delete" of aged data) and create new ones for new data. Will the "usePowerOf2Sizes" also make the reusing of the disk space more efficient for these collections?
          Hide
          jblackburn James Blackburn added a comment -

          I wonder if this is the root cause of:
          SERVER-8078

          Note, for the example script given, useProwerOf2Sizes seems to make the space leak worse rather than better.

          Show
          jblackburn James Blackburn added a comment - I wonder if this is the root cause of: SERVER-8078 Note, for the example script given, useProwerOf2Sizes seems to make the space leak worse rather than better.

            People

            • Votes:
              19 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:
                Days since reply:
                2 years, 11 weeks, 1 day ago
                Date of 1st Reply: