Core Server
  1. Core Server
  2. SERVER-4000

command to change shard key of a collection

    Details

    • Type: New Feature New Feature
    • Status: Open Open
    • Priority: Major - P3 Major - P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: planned but not scheduled
    • Component/s: Sharding
    • Labels:
      None
    • Backport:
      No
    • # Replies:
      7
    • Last comment by Customer:
      false

      Description

      Changing shard keys is fundamentally very expensive, but a helper to do this would be useful. The main thing needed would be to do the operation with good parallelism.

      first cut might require the source collection be read only during the operation.

      might do something like

      • measure what the new distribution would be like by looking at a sampled set of records from the originating collection
      • presplit based on statistics above
      • cluster wide copy of data from src to dest collection
      • build the index(es) for dest after the copy to make things as fast as possible

      i suppose this is just a better version of cloneCollection which we'll want anyway.

        Issue Links

          Activity

          Hide
          John Crenshaw
          added a comment -

          Not needed very often, but on rare occasion it can become really important. I think the most likely use of this may be to fix a collection that was sharded on a linear key (like a MongoId) and needs to be changed to shard against something else (like a hash of that key).

          IMO supporting writes is probably vital. Sharding isn't likely to be used in environments where it would be OK for writes to fail for a prolonged period of time. If triggers are done first you could easily ensure that new data is copied over. The process might be:

          • Create a new sharded destination collection
          • Guess at distribution and presplit
          • Add insert/update/delete triggers on the source collection to clone any changes to the destination collection
          • Copy the data
          • Add the indexes to the destination collection

          Also, wondering about some stuff at the end:

          • Sanity checks? (verify that record counts match, possibly other checks?)
          • Rename the collections dest>>src? (lock needed to make this atomic?)
          • Remove the old source collection or not?
          Show
          John Crenshaw
          added a comment - Not needed very often, but on rare occasion it can become really important. I think the most likely use of this may be to fix a collection that was sharded on a linear key (like a MongoId) and needs to be changed to shard against something else (like a hash of that key). IMO supporting writes is probably vital. Sharding isn't likely to be used in environments where it would be OK for writes to fail for a prolonged period of time. If triggers are done first you could easily ensure that new data is copied over. The process might be: Create a new sharded destination collection Guess at distribution and presplit Add insert/update/delete triggers on the source collection to clone any changes to the destination collection Copy the data Add the indexes to the destination collection Also, wondering about some stuff at the end: Sanity checks? (verify that record counts match, possibly other checks?) Rename the collections dest>>src? (lock needed to make this atomic?) Remove the old source collection or not?
          Hide
          Mike Hobbs
          added a comment -

          A first pass at providing this functionality might be to allow a shard key to be refined. That is, additional fields could be added to an existing key, which would allow more granularity along the existing keys.

          We have sometimes discovered that our collections grow in unexpected ways and that a shard key no longer fits in a 64MB chunk. We can currently increase the max chunk size, but ideally, we'd like to redefine the shard key to make it more granular.

          Show
          Mike Hobbs
          added a comment - A first pass at providing this functionality might be to allow a shard key to be refined. That is, additional fields could be added to an existing key, which would allow more granularity along the existing keys. We have sometimes discovered that our collections grow in unexpected ways and that a shard key no longer fits in a 64MB chunk. We can currently increase the max chunk size, but ideally, we'd like to redefine the shard key to make it more granular.
          Hide
          Eliot Horowitz
          added a comment -

          For making a key more granular we should probably add a new case as that's a lot easier to do since it requires no data movement.

          Show
          Eliot Horowitz
          added a comment - For making a key more granular we should probably add a new case as that's a lot easier to do since it requires no data movement.
          Hide
          Mike Hobbs
          added a comment - - edited

          One option that doesn't require write locks, triggers, or 2X storage capacity:

          The sharding config can have 2 shard configurations per collection - a previous configuration and a current configuration. When a collection is re-keyed, the previous configuration is frozen so that no more splits or migrations are performed based on the previous key. The collection is then re-split and moved about based on the new key. As the data is re-distributed, the previous configuration is not updated - it remains frozen.

          When a collection has multiple shard configurations, the mongos processes would distribute operations out to several shards based on both the previous and current configurations. A record will exist in its old shard if it has not yet moved - or it could potentially exist in a new shard if it has been moved. (There is an issue here when non-multi updates, upserts, and inserts do not contain both the previous and the new shard key)

          When all chunks are migrated based on the new key, the previous configuration is removed.

          Show
          Mike Hobbs
          added a comment - - edited One option that doesn't require write locks, triggers, or 2X storage capacity: The sharding config can have 2 shard configurations per collection - a previous configuration and a current configuration. When a collection is re-keyed, the previous configuration is frozen so that no more splits or migrations are performed based on the previous key. The collection is then re-split and moved about based on the new key. As the data is re-distributed, the previous configuration is not updated - it remains frozen. When a collection has multiple shard configurations, the mongos processes would distribute operations out to several shards based on both the previous and current configurations. A record will exist in its old shard if it has not yet moved - or it could potentially exist in a new shard if it has been moved. (There is an issue here when non-multi updates, upserts, and inserts do not contain both the previous and the new shard key) When all chunks are migrated based on the new key, the previous configuration is removed.
          Hide
          Eliot Horowitz
          added a comment -

          You still have to handle data changes.
          If I change the shard key from (a) to (b) the documents in each chunk will be totally different.
          The meta data changes aren't so bad - its the actual in flight data that's hard.

          Show
          Eliot Horowitz
          added a comment - You still have to handle data changes. If I change the shard key from (a) to (b) the documents in each chunk will be totally different. The meta data changes aren't so bad - its the actual in flight data that's hard.
          Hide
          Mike Hobbs
          added a comment -

          Regarding my earlier comment from Mar 13:

          I'm sure there are subtleties to the implementation of chunks that I'm not fully appreciating, but if there was a special move operation, though, that could move individual records from one shard to another, it would facilitate the migration of records from one key to another. Perhaps such a move operation could be the first step in solving this problem?

          Again, I am ignorant about the implementation of chunks, so forgive me if such an idea is naive.

          Show
          Mike Hobbs
          added a comment - Regarding my earlier comment from Mar 13: I'm sure there are subtleties to the implementation of chunks that I'm not fully appreciating, but if there was a special move operation, though, that could move individual records from one shard to another, it would facilitate the migration of records from one key to another. Perhaps such a move operation could be the first step in solving this problem? Again, I am ignorant about the implementation of chunks, so forgive me if such an idea is naive.
          Hide
          Eliot Horowitz
          added a comment -

          Sadly not that simple.

          Lets say you have 100 shards, and you want to change shard key from (a) to (b).
          If those keys aren't related, then 99% of the data has to be moved.

          So, question is how to do you move data while maintaining state and keeping data live.

          More later if your curious...

          Show
          Eliot Horowitz
          added a comment - Sadly not that simple. Lets say you have 100 shards, and you want to change shard key from (a) to (b). If those keys aren't related, then 99% of the data has to be moved. So, question is how to do you move data while maintaining state and keeping data live. More later if your curious...

            People

            • Votes:
              12 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Days since reply:
                1 year, 51 weeks, 2 days ago
                Date of 1st Reply: