[SERVER-6605] How to add a replica set's member into the sharding's shards Created: 26/Jul/12  Updated: 15/Aug/12  Resolved: 01/Aug/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.6
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Jianfeng Xu Assignee: William Zola
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongos.log    
Issue Links:
Duplicate
is duplicated by SERVER-6680 Replica seed list in the config serve... Closed
Operating System: ALL
Participants:

 Description   

Hi, a sharding cluster has two shard groups:

shards:

{ "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }

The shard shard1 is consisted by three nodes: two mongod nodes and a arbiter node. Then, a new secondary nodes was added. Like below.

shard1:
PRIMARY> rs.config()
{
"_id" : "youninameshard1",
"version" : 3,
"members" : [

{ "_id" : 0, "host" : "10.136.16.67:27032", "priority" : 2 }

,

{ "_id" : 1, "host" : "10.136.16.68:27032" }

,

{ "_id" : 2, "host" : "10.136.16.68:37032", "arbiterOnly" : true }

,

{ "_id" : 3, "host" : "10.136.16.73:27032" }

]
}

Now, how to add the member 3 node into the shard shard1?



 Comments   
Comment by William Zola [ 01/Aug/12 ]

Hi Jianfeng!

Thanks for following-up. I'm having a hard time understanding your last response.

1) You're correct: the output of 'sh.status()' is the result of querying the data in the config server, and the 'config.shards' collection in particular.

2) You're correct: all three config servers must be up and running properly in order for there to be any changes to the metadata of the sharded cluster. If one of the config servers is down, then the metadata for the cluster becomes read-only.

This means that if one of the config servers goes down, all chunk splitting and migration stops.

This also means that if one of the config servers goes down, MongoDB cannot make changes to the "seed list" in the config server metadata.

3) You are also correct that you can continue to write data to the cluster, even if the metadata is read-only.

This behavior is the result of deliberate design decisions by the MongoDB architects. You can read here for more information about those decisions:

4) It sounds like you're saying that your problem was due to one of your config servers being down. Is this the case? It also sounds like you're saying that bringing up the full set of config servers fixed your problem. Please let me know if my understanding is correct.

Please let me know if you have further questions.

-William

Comment by Randolph Tan [ 01/Aug/12 ]

Hi,

I have created a new ticket (SERVER-6680) that describes the bug that you are having. For the meantime, if you want the config database to get updated, either:

1. Make sure all config servers are up and healthy.
or
2. Update the document inside config.shards collection to reflect the latest seed list (and you have to do this on all config servers). This option is a sure fire way to do it but can be a little bit risky if you made a mistake.

Comment by Jianfeng Xu [ 01/Aug/12 ]

Thanks!

I guess that the mongos can listen a new node that is added into a mongod's replica set(refers log of mongos), but the config servers metadata are not synced to all of mongods and the collection "config.shards" data is error. So when using "sh.status()" to query, it just find the data of "config.shards".

This case is easy to reproduct.

It's a big problem when the config servers' metadata are not synced to all of mongods. For example, when a config server goes down, other of config servers is read-only, but the whole sharding is write-read(refers mongodb document) and APP write request is alway writing data to mongod.

Comment by Randolph Tan [ 31/Jul/12 ]

There seems to be nothing wrong based on the mongos logs. Would it be possible to provide the verbose logs of all the config servers? You can set the logging to a higher verbosity without shutting down the server by using the setParam command:

db.adminCommand({ setParameter: 1, logLevel: 5 });

and change it back later to the original value.

Comment by Jianfeng Xu [ 31/Jul/12 ]

Hi, Randolph

1. 3 config servers. You can look at the previous detail comments i chat with William.
2. I posted a mongos node's log that mongos is starting with -vvvvv parameter. Pls download the attachment file mongs.log.

Thanks

Comment by Randolph Tan [ 30/Jul/12 ]

Hi,

Couple of questions/requests:

1. How many config servers do you have?
2. Can you try running mongos with a higher verbosity by passing -vvvvv as the parameter and post the full logs?

Thanks!

Comment by Jianfeng Xu [ 30/Jul/12 ]

Hi William!

Please look at the output of rs.status(), the "10.136.16.73:27032" node is in the SECONDARY status, but it's not work well and the sh.status() is not updated automatically. I also restarted all of mongos servers and config servers. I check all mongos logs and the mongos servers have successed to connect to "10.136.16.73:27032" node before they start. I guess that the config servers' metadata are not updated correctly.

Any ideas? Thanks!

##rs.status()############################################################
PRIMARY> rs.status()
{
"set" : "shard1",
"date" : ISODate("2012-07-30T05:24:31Z"),
"myState" : 1,
"syncingTo" : "10.136.16.68:27032",
"members" : [
{
"_id" : 0,
"name" : "10.136.16.67:27032",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" :

{ "t" : 1343616819000, "i" : 1 }

,
"optimeDate" : ISODate("2012-07-30T02:53:39Z"),
"self" : true
},
{
"_id" : 1,
"name" : "10.136.16.68:27032",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 9756,
"optime" :

{ "t" : 1343616819000, "i" : 1 }

,
"optimeDate" : ISODate("2012-07-30T02:53:39Z"),
"lastHeartbeat" : ISODate("2012-07-30T05:24:29Z"),
"pingMs" : 0
},
{
"_id" : 2,
"name" : "10.136.16.68:37032",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 9758,
"optime" :

{ "t" : 0, "i" : 0 }

,
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2012-07-30T05:24:29Z"),
"pingMs" : 0
},
{
"_id" : 3,
"name" : "10.136.16.73:27032",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 9634,
"optime" :

{ "t" : 1343616819000, "i" : 1 }

,
"optimeDate" : ISODate("2012-07-30T02:53:39Z"),
"lastHeartbeat" : ISODate("2012-07-30T05:24:29Z"),
"pingMs" : 0
}
],
"ok" : 1
}
###sh.status()#################################################################
mongos> sh.status()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" }

d_name.t_name chunks:
shard2 244
shard1 245
too many chunks to print, use verbose if you want to force print

{ "_id" : "test", "partitioned" : false, "primary" : "shard1" }

###mongos log###################################################################
Mon Jul 30 13:53:28 [Balancer] SyncClusterConnection connecting to [10.136.20.59:20000]
Mon Jul 30 13:53:28 [Balancer] starting new replica set monitor for replica set shard1 with seed of 10.136.16.67:27032,10.136.16.68:27032
Mon Jul 30 13:53:28 [Balancer] successfully connected to seed 10.136.16.67:27032 for replica set shard1
Mon Jul 30 13:53:28 [Balancer] changing hosts to

{ 0: "10.136.16.67:27032", 1: "10.136.16.73:27032", 2: "10.136.16.68:27032" }

from shard1/
Mon Jul 30 13:53:28 [Balancer] trying to add new host 10.136.16.67:27032 to replica set shard1
Mon Jul 30 13:53:28 [Balancer] successfully connected to new host 10.136.16.67:27032 in replica set shard1
Mon Jul 30 13:53:28 [Balancer] trying to add new host 10.136.16.68:27032 to replica set shard1
Mon Jul 30 13:53:28 [Balancer] successfully connected to new host 10.136.16.68:27032 in replica set shard1
Mon Jul 30 13:53:28 [Balancer] trying to add new host 10.136.16.73:27032 to replica set shard1
Mon Jul 30 13:53:28 [Balancer] successfully connected to new host 10.136.16.73:27032 in replica set shard1
Mon Jul 30 13:53:28 [Balancer] replica set monitor for replica set shard1 started, address is shard1/10.136.16.67:27032,10.136.16.68:27032,10.136.16.73:27032
Mon Jul 30 13:53:28 [ReplicaSetMonitorWatcher] starting
Mon Jul 30 13:53:28 [Balancer] starting new replica set monitor for replica set shard2 with seed of 10.136.16.69:27032,10.136.16.70:27032
Mon Jul 30 13:53:28 [Balancer] successfully connected to seed 10.136.16.69:27032 for replica set shard2
Mon Jul 30 13:53:28 [Balancer] changing hosts to

{ 0: "10.136.16.69:27032", 1: "10.136.16.70:27032" }

from shard2/
Mon Jul 30 13:53:28 [Balancer] trying to add new host 10.136.16.69:27032 to replica set shard2
Mon Jul 30 13:53:28 [Balancer] successfully connected to new host 10.136.16.69:27032 in replica set shard2
Mon Jul 30 13:53:28 [Balancer] trying to add new host 10.136.16.70:27032 to replica set shard2
Mon Jul 30 13:53:28 [Balancer] successfully connected to new host 10.136.16.70:27032 in replica set shard2
Mon Jul 30 13:53:28 [Balancer] replica set monitor for replica set shard2 started, address is shard2/10.136.16.69:27032,10.136.16.70:27032
Mon Jul 30 13:53:28 [Balancer] config servers and shards contacted successfully

Comment by William Zola [ 27/Jul/12 ]

Hi Jianfeng!

All of that looks good. As I said before, the sh.status() should update automatically. Can you please send me the output of rs.status() from one of the nodes in shard1. It is possible that your new host is not yet in SECONDARY status, and the output of this command will check for that.

Please let me know. Have a great day!

-William

Comment by Jianfeng Xu [ 27/Jul/12 ]

Thanks!

  • MongoDB version: mongodb-linux-x86_64-2.0.6
  • PRIMARY> db.isMaster()
    {
    "setName" : "shard1",
    "ismaster" : true,
    "secondary" : false,
    "hosts" : [
    "10.136.16.67:27032",
    "10.136.16.73:27032",
    "10.136.16.68:27032"
    ],
    "arbiters" : [
    "10.136.16.68:37032"
    ],
    "primary" : "10.136.16.67:27032",
    "me" : "10.136.16.67:27032",
    "maxBsonObjectSize" : 16777216,
    "ok" : 1
    }
  • There are three config servers in this sharding: 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000. All of nodes are online and active. I show some informations from mongos log.

Fri Jul 27 09:09:39 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Fri Jul 27 09:09:39 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2057:30000:1343047408:1804289383', sleeping for 30000ms

  • Almost one day. "Thu Jul 26 15:07:26", i added it. Now "Fri Jul 27 09:24:16", the sh.status()'s output is like this. It is not update by itself.

[root@mongodb-1667 ~]# date
Fri Jul 27 09:24:16 CST 2012
[root@mongodb-1667 ~]# mongo --host=10.136.20.57:30000
MongoDB shell version: 2.0.6
connecting to: 10.136.20.57:30000/test
mongos> sh.status()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" }

d_name.t_name chunks:
shard1 243
shard2 243
too many chunks to print, use verbose if you want to force print

{ "_id" : "test", "partitioned" : false, "primary" : "shard1" }

– popgo

Comment by William Zola [ 26/Jul/12 ]

Hi Jianfeng!

I'll need additional information to diagnose this problem. Please let me know the following:

  • What version of MongoDB are you running?
  • Please send me the output of db.isMaster() when you are connected to a node in the shard1 replica set
  • How many config servers are you running? Are they all currently active?
  • How long did you wait between adding the new node and running sh.status()?

I'd like to help you solve this problem. I can't move forward without this information.

-William

Comment by Jianfeng Xu [ 26/Jul/12 ]

Thanks for your reply.

Yes, u can look the output of sh.status().

mongos> sh.status()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" }

d_name.t_name chunks:
shard1 243
shard2 243
too many chunks to print, use verbose if you want to force print

{ "_id" : "test", "partitioned" : false, "primary" : "shard1" }
Comment by Randolph Tan [ 26/Jul/12 ]

It should be able to update itself over time. Is it not in your case?

Generated at Thu Feb 08 03:12:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.