Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-2271

Increase initial sharding limit to 256gb

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0-rc0
    • Component/s: Sharding
    • Labels:
      None

      Description

      There's an arbitrary limit on the size of a pre-existing collection that we're split into chunks. Alberto tells me this limit should be revisited, could possibly be removed altogether. Mostly just creating this issue to allow watchers to track progress.

        Issue Links

          Activity

          Hide
          kristina Kristina Chodorow (Inactive) added a comment -

          The constant should be upped for 1.8.0. It's ridiculously low (you can't chunk a collection larger than 16GBs) and shardCollection will look through the entire index regardless, which is a huge waste of resources.

          Show
          kristina Kristina Chodorow (Inactive) added a comment - The constant should be upped for 1.8.0. It's ridiculously low (you can't chunk a collection larger than 16GBs) and shardCollection will look through the entire index regardless, which is a huge waste of resources.
          Hide
          eliot Eliot Horowitz added a comment -

          Try making it 16x larger, and spin up a collection with 200gb, the shard and see what happens.
          If all looks good, go ahead and commit

          Show
          eliot Eliot Horowitz added a comment - Try making it 16x larger, and spin up a collection with 200gb, the shard and see what happens. If all looks good, go ahead and commit
          Hide
          auto auto (Inactive) added a comment -

          Author:

          {u'login': u'kchodorow', u'name': u'Kristina Chodorow', u'email': u'kristina@10gen.com'}

          Message: increase number of initial chunk splits allowed to fit 256GB of data SERVER-2271
          https://github.com/mongodb/mongo/commit/a9e3511eda147d42f79203ad6e181e789c632c49

          Show
          auto auto (Inactive) added a comment - Author: {u'login': u'kchodorow', u'name': u'Kristina Chodorow', u'email': u'kristina@10gen.com'} Message: increase number of initial chunk splits allowed to fit 256GB of data SERVER-2271 https://github.com/mongodb/mongo/commit/a9e3511eda147d42f79203ad6e181e789c632c49
          Hide
          eliot Eliot Horowitz added a comment -

          Note, one way around the limit is increasing chunk size temporarily.

          Show
          eliot Eliot Horowitz added a comment - Note, one way around the limit is increasing chunk size temporarily.
          Hide
          plasma Andrew Armstrong added a comment -

          I noticed on http://www.mongodb.org/display/DOCS/Sharding+FAQ that there is no mention of the limits to being able to go from an unsharded->sharded collection.

          Having read the docs a while ago I would have thought I was safe being unsharded until my data size grew too big; but if there is a limit to where I can transition to a sharded collection that may be worth noting in the docs?

          Cheers

          Show
          plasma Andrew Armstrong added a comment - I noticed on http://www.mongodb.org/display/DOCS/Sharding+FAQ that there is no mention of the limits to being able to go from an unsharded->sharded collection. Having read the docs a while ago I would have thought I was safe being unsharded until my data size grew too big; but if there is a limit to where I can transition to a sharded collection that may be worth noting in the docs? Cheers
          Hide
          eliot Eliot Horowitz added a comment -
          Show
          eliot Eliot Horowitz added a comment - @andrew, its on this page: http://www.mongodb.org/display/DOCS/Sharding+Limits
          Hide
          colinhowe Colin Howe added a comment -

          Is this a limit on the data size or the combined data/index size?

          Show
          colinhowe Colin Howe added a comment - Is this a limit on the data size or the combined data/index size?
          Hide
          kristina Kristina Chodorow (Inactive) added a comment -

          Just the data size.

          Show
          kristina Kristina Chodorow (Inactive) added a comment - Just the data size.
          Hide
          asya Asya Kamsky added a comment -

          The code shows 512GB limit for 64MB default chunk size (max split points 8192) but the docs say 256GB still, opened DOCS-1265 to fix.

          Show
          asya Asya Kamsky added a comment - The code shows 512GB limit for 64MB default chunk size (max split points 8192) but the docs say 256GB still, opened DOCS-1265 to fix.

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: