Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17166

balancer move failed

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical - P2 Critical - P2
    • None
    • None
    • None
    • None
    • ALL

    Description

      hi there

      We just created 3 shards(R7, R8, R9) and added into our system. we are hoping mongodb will do the data balance. but it failed.

      mongos> db.printShardingStatus()
      --- Sharding Status ---
        sharding version: {
              "_id" : 1,
              "version" : 4,
              "minCompatibleVersion" : 4,
              "currentVersion" : 5,
              "clusterId" : ObjectId("542119996fb1141f1f3ce486")
      }
        shards:
              {  "_id" : "R1",  "host" : "R1/mongo40.seoclarity.net:40001,mongo42.seoclarity.net:40001" }
              {  "_id" : "R2",  "host" : "R2/mongo40.seoclarity.net:40002,mongo44.seoclarity.net:40002" }
              {  "_id" : "R3",  "host" : "R3/mongo42.seoclarity.net:40003,mongo44.seoclarity.net:40003" }
              {  "_id" : "R4",  "host" : "R4/mongo46.seoclarity.net:40004,mongo48.seoclarity.net:40004" }
              {  "_id" : "R5",  "host" : "R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005" }
              {  "_id" : "R6",  "host" : "R6/mongo48.seoclarity.net:40006,mongo50.seoclarity.net:40006" }
              {  "_id" : "R7",  "host" : "R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007" }
              {  "_id" : "R8",  "host" : "R8/mongo54.seoclarity.net:40008,mongo56.seoclarity.net:40008" }
              {  "_id" : "R9",  "host" : "R9/mongo52.seoclarity.net:40009,mongo54.seoclarity.net:40009" }
        databases:
              {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
              {  "_id" : "mongo_seoclarity_rankcheck",  "partitioned" : true,  "primary" : "R1" }
                      mongo_seoclarity_rankcheck.keyword_rank_entity
                              shard key: { "kName" : 1, "date" : 1, "engine" : 1, "language" : 1 }
                              chunks:
                                      R7      11
                                      R9      11
                                      R8      11
                                      R1      11424
                                      R5      11425
                                      R4      11424
                                      R6      11425
                                      R2      11424
                                      R3      11424
                              too many chunks to print, use verbose if you want to force print
              {  "_id" : "test",  "partitioned" : false,  "primary" : "R1" }
              {  "_id" : "mongo_google_pagespeed",  "partitioned" : false,  "primary" : "R2" }
              {  "_id" : "actonia_crawler_tracking",  "partitioned" : false,  "primary" : "R2" }
              {  "_id" : "seoclarity_googlewebmaster",  "partitioned" : false,  "primary" : "R2" }
              {  "_id" : "seoclarity_trend",  "partitioned" : false,  "primary" : "R5" }
              {  "_id" : "keyword_rank_entity",  "partitioned" : false,  "primary" : "R5" }
              {  "_id" : "mongo_seoclarity_rankcheckl",  "partitioned" : false,  "primary" : "R5" }
       
      mongos>

      log file

      2015-02-03T14:39:35.325-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
      2015-02-03T14:39:35.326-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
      2015-02-03T14:39:35.469-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.
      2015-02-03T14:39:41.653-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' acquired, ts : 54d1320dd8e90ef760f1ef25
      2015-02-03T14:39:42.112-0600 [Balancer]  ns: mongo_seoclarity_rankcheck.keyword_rank_entity going to move { _id: "mongo_seoclarity_rankcheck.keyword_rank_entity-kName_"%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+...", lastmod: Timestamp 13904000|1, lastmodEpoch: ObjectId('54211ac5d8e90ef760ddd1de'), ns: "mongo_seoclarity_rankcheck.keyword_rank_entity", min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }, shard: "R5" } from: R5 to: R7 tag []
      2015-02-03T14:39:42.124-0600 [Balancer] moving chunk ns: mongo_seoclarity_rankcheck.keyword_rank_entity moving ( ns: mongo_seoclarity_rankcheck.keyword_rank_entity, shard: R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005, lastmod: 13904|1||000000000000000000000000, min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }) R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005 -> R7:R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007
      2015-02-03T14:39:42.132-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
      2015-02-03T14:39:42.133-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
      2015-02-03T14:39:42.201-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.
      2015-02-03T14:39:45.846-0600 [conn13863893] end connection 127.0.0.1:58464 (668 connections now open)
      2015-02-03T14:39:48.354-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' acquired, ts : 54d13214d8e90ef760f1ef26
      2015-02-03T14:39:48.808-0600 [Balancer]  ns: mongo_seoclarity_rankcheck.keyword_rank_entity going to move { _id: "mongo_seoclarity_rankcheck.keyword_rank_entity-kName_"%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+...", lastmod: Timestamp 13904000|1, lastmodEpoch: ObjectId('54211ac5d8e90ef760ddd1de'), ns: "mongo_seoclarity_rankcheck.keyword_rank_entity", min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }, shard: "R5" } from: R5 to: R7 tag []
      2015-02-03T14:39:48.820-0600 [Balancer] moving chunk ns: mongo_seoclarity_rankcheck.keyword_rank_entity moving ( ns: mongo_seoclarity_rankcheck.keyword_rank_entity, shard: R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005, lastmod: 13904|1||000000000000000000000000, min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }) R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005 -> R7:R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007
      2015-02-03T14:39:48.829-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
      2015-02-03T14:39:48.829-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
      2015-02-03T14:39:48.902-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.

      Please help to check asap

      Thanks
      Alps

      Attachments

        Activity

          People

            Unassigned Unassigned
            mgandhi@seoclarity.net Dev
            Dev
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: