[SERVER-19495] Can't split hash sharded collection with empty chunks Created: 20/Jul/15  Updated: 25/Aug/15  Resolved: 25/Aug/15

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Aaron Westendorf Assignee: Randolph Tan
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

In a 2.6.9 cluster, start with a sharded collection that has about 580 documents with an average size of 320 bytes. Use hashed sharding and ensure that it starts with 1 chunk. The following python is our code to split the chunks:

def split_chunks(namespace, host='localhost', dry_run=True, min_chunk=None, max_chunk=None):
  '''
  Split chunks of a <db>.<collection> namespace. Defaults to a dry run with no
  changes. If making changes, will first stop, and lastly restart, the balancer
  ONLY when running on localhost via salt-call (recommended).
 
  :host: Defaults to localhost. Balancer support only works in this case
  :dry_run: Defaults to true. If false, changes will be made
  :min_chunk: Optional json doc that defines the starting chunking from which to
              start splitting. Must be an exact match on the start of a chunk.
              Mostly useful when restarting a previous split
              operation that was cancelled or interrupted.
  :max_chunk: Optional json doc that defines the end chunking from which to
              finish splitting. Must be an exact match on the end of a chunk.
  '''
  result = {'result':True, 'data':[]}
  mongos = pymongo.MongoClient(host=host)
  config_host = mongos['admin'].command('getCmdLineOpts')['parsed']\
    ['sharding']['configDB'].split(',')[0]
  mongoc = pymongo.MongoClient(host=config_host, port=27019)
 
  if not dry_run and host == 'localhost':
    res = stop_balancer()
    result['data'].append( res['data'] )
    result['result'] = res['result']
    if not result['result']:
      return result
 
  chunks = mongoc['config']['chunks'].find({'ns':namespace})
  chunk_count = chunks.count()
 
  # Calculate all of the splits to be sure we don't timeout the chunks cursor
  splits = []
  for chunk in chunks:
    if min_chunk:
      if chunk['min'] == min_chunk:
        min_chunk = None
      else:
        continue
    if max_chunk:
      if chunk['max'] == max_chunk:
        splits.append([chunk['min'], chunk['max']])
        break
    splits.append([chunk['min'], chunk['max']])
 
  log.info('Splitting {}/{} chunks'.format(len(splits), chunk_count))
  for s in splits:
    if dry_run:
      log.info('  dry_run split {}'.format(s))
    else:
      log.info('  split {}'.format(s))
      mongos['admin'].command( {'split':namespace, 'bounds':s} )
  if dry_run:
    result['data'].append('dry_run {} chunks'.format(len(splits)))
  else:
    result['data'].append('split {} chunks'.format(len(splits)))
 
  if not dry_run and host == 'localhost':
    res = start_balancer()
    result['data'].append( res['data'] )
    result['result'] = res['result']
 
  return result

Participants:

 Description   

We have an existing collection with a small number of records that will soon see a lot more. It was already sharded on a hashed application guid. Our goal is equivalent to a pre-split of an empty collection; we want to match the number of chunks to the new expected use of this collection.

We're using a splitting routine that lists the chunks in the config server, and for each chunk, calls split with the bounds. We were able to split this collection until we hit a point where it seems that there's an empty chunk which we can't split.

2015-07-20T16:50:21.009+0000 [conn121938] splitting chunk [{ guid: 4611686018427387902 },{ guid: 4625894529193607559 }) in collection game_production.objects on shard game-mongodb-cluster-1-data-6
2015-07-20T16:50:21.009+0000 [conn121938] want to split chunk, but can't find split point chunk ns: game_production.objects, shard: game-mongodb-cluster-1-data-6:game-mongodb-cluster-1-data-6/game-mongodb-cluster-1-data-6-1:27018,game-mongodb-cluster-1-data-6-2:27018,game-mongodb-cluster-1-data-6-3:27018, lastmod: 8|434||000000000000000000000000, min: { guid: 4611686018427387902 }, max: { guid: 4625894529193607559 } got: <empty>

The only correlated log we can find across all mongo processes is this from the primary in that shard:

2015-07-20T15:58:49.756+0000 [conn240524] request split points lookup for chunk game_production.objects { : 4611686018427387902 } -->> { : 4625894529193607559 }
2015-07-20T15:58:49.756+0000 [conn240524] splitVector doing another cycle because of force, keyCount now: 0
2015-07-20T15:58:49.756+0000 [conn240524] warning: chunk is larger than 17984 bytes because of key { guid: 4613685088596591651 }

Our expectation is that, even if the chunk is empty, the key range can be split without a problem.



 Comments   
Comment by Ramon Fernandez Marina [ 10/Aug/15 ]

aaron.westendorf, we haven't heard back from you for some time. If this is still an issue for you can you please answer Randolph's questions above?

Thanks,
Ramón.

Comment by Randolph Tan [ 28/Jul/15 ]

Hi,

Can you clarify by what you mean by it didn't do anything? Did you get an error from it? I have also contacted the docs team to revise the "do not use middle" for hashed shard keys.

Because a hashed chunk is actually defined by the range of hashed integers that will end up in the chunk, I expect that it should split that range, such that range(a, b) splits to range(a, a+(b-a)/2) , range(a+(b-a)/2, b)

That is not how split with bounds work right now. The shard key of the median is selected as the split point. So, if there are no documents in the chunk range, then there will be no median to select from.

Thanks!

Comment by Aaron Westendorf [ 27/Jul/15 ]

We tried middle, it didn't do anything on a hash shard key. In the documentation you linked to:

To create a split for a collection that uses a hashed shard key, use the bounds parameter. Do not use the middle parameter for this purpose.

Because a hashed chunk is actually defined by the range of hashed integers that will end up in the chunk, I expect that it should split that range, such that range(a, b) splits to range(a, a+(b-a)/2) , range(a+(b-a)/2, b).

Comment by Randolph Tan [ 27/Jul/15 ]

Hi,

The "bounds" field is similar to the "find" field that it needs some data in the chunk because the split point is determined based on the number of documents in the chunk. If you want to perform a split regardless of whether a chunk is empty or not, you should use the "middle" field. Also see: http://docs.mongodb.org/manual/reference/command/split/#considerations

Generated at Thu Feb 08 03:51:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.