[SERVER-17166] balancer move failed Created: 03/Feb/15  Updated: 03/Feb/15  Resolved: 03/Feb/15

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Dev Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:
Dev

 Description   

hi there

We just created 3 shards(R7, R8, R9) and added into our system. we are hoping mongodb will do the data balance. but it failed.

mongos> db.printShardingStatus()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "version" : 4,
        "minCompatibleVersion" : 4,
        "currentVersion" : 5,
        "clusterId" : ObjectId("542119996fb1141f1f3ce486")
}
  shards:
        {  "_id" : "R1",  "host" : "R1/mongo40.seoclarity.net:40001,mongo42.seoclarity.net:40001" }
        {  "_id" : "R2",  "host" : "R2/mongo40.seoclarity.net:40002,mongo44.seoclarity.net:40002" }
        {  "_id" : "R3",  "host" : "R3/mongo42.seoclarity.net:40003,mongo44.seoclarity.net:40003" }
        {  "_id" : "R4",  "host" : "R4/mongo46.seoclarity.net:40004,mongo48.seoclarity.net:40004" }
        {  "_id" : "R5",  "host" : "R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005" }
        {  "_id" : "R6",  "host" : "R6/mongo48.seoclarity.net:40006,mongo50.seoclarity.net:40006" }
        {  "_id" : "R7",  "host" : "R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007" }
        {  "_id" : "R8",  "host" : "R8/mongo54.seoclarity.net:40008,mongo56.seoclarity.net:40008" }
        {  "_id" : "R9",  "host" : "R9/mongo52.seoclarity.net:40009,mongo54.seoclarity.net:40009" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "mongo_seoclarity_rankcheck",  "partitioned" : true,  "primary" : "R1" }
                mongo_seoclarity_rankcheck.keyword_rank_entity
                        shard key: { "kName" : 1, "date" : 1, "engine" : 1, "language" : 1 }
                        chunks:
                                R7      11
                                R9      11
                                R8      11
                                R1      11424
                                R5      11425
                                R4      11424
                                R6      11425
                                R2      11424
                                R3      11424
                        too many chunks to print, use verbose if you want to force print
        {  "_id" : "test",  "partitioned" : false,  "primary" : "R1" }
        {  "_id" : "mongo_google_pagespeed",  "partitioned" : false,  "primary" : "R2" }
        {  "_id" : "actonia_crawler_tracking",  "partitioned" : false,  "primary" : "R2" }
        {  "_id" : "seoclarity_googlewebmaster",  "partitioned" : false,  "primary" : "R2" }
        {  "_id" : "seoclarity_trend",  "partitioned" : false,  "primary" : "R5" }
        {  "_id" : "keyword_rank_entity",  "partitioned" : false,  "primary" : "R5" }
        {  "_id" : "mongo_seoclarity_rankcheckl",  "partitioned" : false,  "primary" : "R5" }
 
mongos>

log file

2015-02-03T14:39:35.325-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
2015-02-03T14:39:35.326-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
2015-02-03T14:39:35.469-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.
2015-02-03T14:39:41.653-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' acquired, ts : 54d1320dd8e90ef760f1ef25
2015-02-03T14:39:42.112-0600 [Balancer]  ns: mongo_seoclarity_rankcheck.keyword_rank_entity going to move { _id: "mongo_seoclarity_rankcheck.keyword_rank_entity-kName_"%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+...", lastmod: Timestamp 13904000|1, lastmodEpoch: ObjectId('54211ac5d8e90ef760ddd1de'), ns: "mongo_seoclarity_rankcheck.keyword_rank_entity", min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }, shard: "R5" } from: R5 to: R7 tag []
2015-02-03T14:39:42.124-0600 [Balancer] moving chunk ns: mongo_seoclarity_rankcheck.keyword_rank_entity moving ( ns: mongo_seoclarity_rankcheck.keyword_rank_entity, shard: R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005, lastmod: 13904|1||000000000000000000000000, min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }) R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005 -> R7:R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007
2015-02-03T14:39:42.132-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
2015-02-03T14:39:42.133-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
2015-02-03T14:39:42.201-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.
2015-02-03T14:39:45.846-0600 [conn13863893] end connection 127.0.0.1:58464 (668 connections now open)
2015-02-03T14:39:48.354-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' acquired, ts : 54d13214d8e90ef760f1ef26
2015-02-03T14:39:48.808-0600 [Balancer]  ns: mongo_seoclarity_rankcheck.keyword_rank_entity going to move { _id: "mongo_seoclarity_rankcheck.keyword_rank_entity-kName_"%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+...", lastmod: Timestamp 13904000|1, lastmodEpoch: ObjectId('54211ac5d8e90ef760ddd1de'), ns: "mongo_seoclarity_rankcheck.keyword_rank_entity", min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }, shard: "R5" } from: R5 to: R7 tag []
2015-02-03T14:39:48.820-0600 [Balancer] moving chunk ns: mongo_seoclarity_rankcheck.keyword_rank_entity moving ( ns: mongo_seoclarity_rankcheck.keyword_rank_entity, shard: R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005, lastmod: 13904|1||000000000000000000000000, min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 }, max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }) R5:R5/mongo46.seoclarity.net:40005,mongo50.seoclarity.net:40005 -> R7:R7/mongo52.seoclarity.net:40007,mongo56.seoclarity.net:40007
2015-02-03T14:39:48.829-0600 [Balancer] moveChunk result: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." }
2015-02-03T14:39:48.829-0600 [Balancer] balancer move failed: { ok: 0.0, errmsg: "error locking distributed lock for migration migrate-{ kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+th..." } from: R5 to: R7 chunk:  min: { kName: "%22%22%22predictably+irrational%2c+revised+and+expanded+edition%3a+the+hidden+forces+that+shape+our+decisions%22%22%22", date: "20140915", engine: 255, language: 1 } max: { kName: "%22%22%22ruth+bell%22%22+%22%22helen+stam%22%22%22", date: "20140906", engine: 1, language: 1 }
2015-02-03T14:39:48.902-0600 [Balancer] distributed lock 'balancer/mongo40.seoclarity.net:40000:1411455633:1804289383' unlocked.

Please help to check asap

Thanks
Alps



 Comments   
Comment by Dev [ 03/Feb/15 ]

never mind, i figured it out. please close this ticket

Thanks
Alps

Generated at Thu Feb 08 03:43:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.