[SERVER-2985] Rebalancing too slow and moveChunk is blocked by balancer lock Created: 22/Apr/11  Updated: 10/May/12  Resolved: 02/Sep/11

Status: Closed
Project: Core Server
Component/s: Admin, Sharding
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Question Priority: Critical - P2
Reporter: John Schulz Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: concurrency, sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS Centos 5.4. Host HW dual socket Nehailm 4 cores, 36GB memory 24 1TB disks in Raid10 configuration. New Shard has 64GB of memory with 12 300 GB disks in Raid 10 configuration.


Attachments: File slow_rebalance_data_for_10Gen.tar    
Participants:

 Description   

We have added a new shard to a 4 shard cluster making it 5 shards. The cluster is under a very light workload. Watching the load balancer it would appear that its going to take 2-3 days to complete rebalancing the shards.

> db.printShardingStatus();
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:
{
"_id" : "repset_a",
"host" : "repset_a/lmdb-m03.mail.aol.com:7312,lmdb-d02.mail.aol.com:7312,lmdb-d01.mail.aol.com:7312"
}
{
"_id" : "repset_b",
"host" : "repset_b/lmdb-d05.mail.aol.com:7312,lmdb-m06.mail.aol.com:7312,lmdb-d04.mail.aol.com:7312"
}
{
"_id" : "repset_c",
"host" : "repset_c/lmdb-d03.mail.aol.com:7312,lmdb-m02.mail.aol.com:7312,lmdb-m01.mail.aol.com:7312"
}
{
"_id" : "repset_d",
"host" : "repset_d/lmdb-d06.mail.aol.com:7312,lmdb-m05.mail.aol.com:7312,lmdb-m04.mail.aol.com:7312"
}
{
"_id" : "repset_e",
"host" : "repset_e/lmdb-d08.mail.aol.com:7312,lmdb-m09.mail.aol.com:7312,lmdb-d07.mail.aol.com:7312"
}
databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "MigOidDB", "partitioned" : true, "primary" : "repset_a" }

MigOidDB.MigOidCol chunks:
repset_e 205
repset_c 1283
repset_a 1283
repset_d 1283
repset_b 1283
too many chunksn to print, use verbose if you want to force print

{ "_id" : "test", "partitioned" : false, "primary" : "repset_a" } { "_id" : "local", "partitioned" : false, "primary" : "repset_a" } { "_id" : "MigOidCol", "partitioned" : false, "primary" : "repset_a" }

We have tried using moveChunk to speed the process up but the load balancer has a "Metadata Lock" on the collection and will not allow us to do a manual moveChunk.

> db.adminCommand({moveChunk : "MigOidDB.MigOidCol", find : {_id : "buggzeeann_30324171"}, to : "repset_e"});
{
"cause" : {
"who" : {
"_id" : "MigOidDB.MigOidCol",
"process" : "lmdb-d02.mail.aol.com:1303133674:1828878975",
"state" : 1,
"ts" : ObjectId("4db174cf7a18929e98aa4f5b"),
"when" : ISODate("2011-04-22T12:30:07.322Z"),
"who" : "lmdb-d02.mail.aol.com:1303133674:1828878975:conn34711:1543737603",
"why" : "migrate-

{ _id: \"ballc21_28406862\" }

"
},
"errmsg" : "the collection's metadata lock is taken",
"ok" : 0
},
"ok" : 0,
"errmsg" : "move failed"
}



 Comments   
Comment by Eliot Horowitz (Inactive) [ 02/Sep/11 ]

Lots of improvements for 1.8.3 if you haven't already tried that.

Comment by Eliot Horowitz (Inactive) [ 29/Apr/11 ]

Sorry didn't have a chance to look at this in the interim.
How much progress has been made in the last few days?

Comment by John Schulz [ 23/Apr/11 ]

Attached the changelog and a verbose printShardingStatus. The acutal server and Mongos logs are very verbose and multi gigs in size. If you can give me some specific ideas as to which logs and what data from those logs you are interested in I will do what I can to collect that data.

1.5 days after shard repset_e and just under 1 day after the rest of the shards were added we have rebalanced 382 out of the 4100+ that we need to rebalance. For several hours yesterday we completely shut down all inserts to the cluster and the rebalancing did not go any faster or slower.

Comment by Eliot Horowitz (Inactive) [ 23/Apr/11 ]

You can only move 1 chunk at a time, so the lock is normal.

Are you sure its "slow" or is bound by disk, etc...

Can you send log files.

Generated at Thu Feb 08 03:01:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.