Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19495

Can't split hash sharded collection with empty chunks

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.9
    • Component/s: Sharding
    • Labels:
      None
    • ALL
    • Hide

      In a 2.6.9 cluster, start with a sharded collection that has about 580 documents with an average size of 320 bytes. Use hashed sharding and ensure that it starts with 1 chunk. The following python is our code to split the chunks:

      def split_chunks(namespace, host='localhost', dry_run=True, min_chunk=None, max_chunk=None):
  '''
  Split chunks of a <db>.<collection> namespace. Defaults to a dry run with no
  changes. If making changes, will first stop, and lastly restart, the balancer
  ONLY when running on localhost via salt-call (recommended).

  :host: Defaults to localhost. Balancer support only works in this case
  :dry_run: Defaults to true. If false, changes will be made
  :min_chunk: Optional json doc that defines the starting chunking from which to
              start splitting. Must be an exact match on the start of a chunk.
              Mostly useful when restarting a previous split
              operation that was cancelled or interrupted.
  :max_chunk: Optional json doc that defines the end chunking from which to
              finish splitting. Must be an exact match on the end of a chunk.
  '''
  result = {'result':True, 'data':[]}
  mongos = pymongo.MongoClient(host=host)
  config_host = mongos['admin'].command('getCmdLineOpts')['parsed']\
    ['sharding']['configDB'].split(',')[0]
  mongoc = pymongo.MongoClient(host=config_host, port=27019)

  if not dry_run and host == 'localhost':
    res = stop_balancer()
    result['data'].append( res['data'] )
    result['result'] = res['result']
    if not result['result']:
      return result

  chunks = mongoc['config']['chunks'].find({'ns':namespace})
  chunk_count = chunks.count()

  # Calculate all of the splits to be sure we don't timeout the chunks cursor
  splits = []
  for chunk in chunks:
    if min_chunk:
      if chunk['min'] == min_chunk:
        min_chunk = None
      else:
        continue
    if max_chunk:
      if chunk['max'] == max_chunk:
        splits.append([chunk['min'], chunk['max']])
        break
    splits.append([chunk['min'], chunk['max']])

  log.info('Splitting {}/{} chunks'.format(len(splits), chunk_count))
  for s in splits:
    if dry_run:
      log.info('  dry_run split {}'.format(s))
    else:
      log.info('  split {}'.format(s))
      mongos['admin'].command( {'split':namespace, 'bounds':s} )
  if dry_run:
    result['data'].append('dry_run {} chunks'.format(len(splits)))
  else:
    result['data'].append('split {} chunks'.format(len(splits)))

  if not dry_run and host == 'localhost':
    res = start_balancer()
    result['data'].append( res['data'] )
    result['result'] = res['result']

  return result
      Show
      In a 2.6.9 cluster, start with a sharded collection that has about 580 documents with an average size of 320 bytes. Use hashed sharding and ensure that it starts with 1 chunk. The following python is our code to split the chunks: def split_chunks(namespace, host= 'localhost' , dry_run= True , min_chunk= None , max_chunk= None ): ''' Split chunks of a <db>.<collection> namespace. Defaults to a dry run with no changes. If making changes, will first stop, and lastly restart, the balancer ONLY when running on localhost via salt-call (recommended). :host: Defaults to localhost. Balancer support only works in this case :dry_run: Defaults to true. If false, changes will be made :min_chunk: Optional json doc that defines the starting chunking from which to start splitting. Must be an exact match on the start of a chunk. Mostly useful when restarting a previous split operation that was cancelled or interrupted. :max_chunk: Optional json doc that defines the end chunking from which to finish splitting. Must be an exact match on the end of a chunk. ''' result = { 'result' : True , 'data' :[]} mongos = pymongo.MongoClient(host=host) config_host = mongos[ 'admin' ].command( 'getCmdLineOpts' )[ 'parsed' ]\ [ 'sharding' ][ 'configDB' ].split( ',' )[0] mongoc = pymongo.MongoClient(host=config_host, port=27019) if not dry_run and host == 'localhost' : res = stop_balancer() result[ 'data' ].append( res[ 'data' ] ) result[ 'result' ] = res[ 'result' ] if not result[ 'result' ]: return result chunks = mongoc[ 'config' ][ 'chunks' ].find({ 'ns' :namespace}) chunk_count = chunks.count() # Calculate all of the splits to be sure we don't timeout the chunks cursor splits = [] for chunk in chunks: if min_chunk: if chunk[ ' min ' ] == min_chunk: min_chunk = None else : continue if max_chunk: if chunk[ ' max ' ] == max_chunk: splits.append([chunk[ ' min ' ], chunk[ ' max ' ]]) break splits.append([chunk[ ' min ' ], chunk[ ' max ' ]]) log.info( 'Splitting {}/{} chunks' . format ( len (splits), chunk_count)) for s in splits: if dry_run: log.info( ' dry_run split {}' . format (s)) else : log.info( ' split {}' . format (s)) mongos[ 'admin' ].command( { 'split' :namespace, 'bounds' :s} ) if dry_run: result[ 'data' ].append( 'dry_run {} chunks' . format ( len (splits))) else : result[ 'data' ].append( 'split {} chunks' . format ( len (splits))) if not dry_run and host == 'localhost' : res = start_balancer() result[ 'data' ].append( res[ 'data' ] ) result[ 'result' ] = res[ 'result' ] return result

      We have an existing collection with a small number of records that will soon see a lot more. It was already sharded on a hashed application guid. Our goal is equivalent to a pre-split of an empty collection; we want to match the number of chunks to the new expected use of this collection.

      We're using a splitting routine that lists the chunks in the config server, and for each chunk, calls split with the bounds. We were able to split this collection until we hit a point where it seems that there's an empty chunk which we can't split.

      2015-07-20T16:50:21.009+0000 [conn121938] splitting chunk [{ guid: 4611686018427387902 },{ guid: 4625894529193607559 }) in collection game_production.objects on shard game-mongodb-cluster-1-data-6
2015-07-20T16:50:21.009+0000 [conn121938] want to split chunk, but can't find split point chunk ns: game_production.objects, shard: game-mongodb-cluster-1-data-6:game-mongodb-cluster-1-data-6/game-mongodb-cluster-1-data-6-1:27018,game-mongodb-cluster-1-data-6-2:27018,game-mongodb-cluster-1-data-6-3:27018, lastmod: 8|434||000000000000000000000000, min: { guid: 4611686018427387902 }, max: { guid: 4625894529193607559 } got: <empty>

      The only correlated log we can find across all mongo processes is this from the primary in that shard:

      2015-07-20T15:58:49.756+0000 [conn240524] request split points lookup for chunk game_production.objects { : 4611686018427387902 } -->> { : 4625894529193607559 }
2015-07-20T15:58:49.756+0000 [conn240524] splitVector doing another cycle because of force, keyCount now: 0
2015-07-20T15:58:49.756+0000 [conn240524] warning: chunk is larger than 17984 bytes because of key { guid: 4613685088596591651 }

      Our expectation is that, even if the chunk is empty, the key range can be split without a problem.

            Assignee:
            randolph@mongodb.com Randolph Tan
            Reporter:
            aaron.westendorf Aaron Westendorf
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: