[SERVER-6605] How to add a replica set's member into the sharding's shards Created: 26/Jul/12 Updated: 15/Aug/12 Resolved: 01/Aug/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.0.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Jianfeng Xu | Assignee: | William Zola |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
Hi, a sharding cluster has two shard groups: shards: { "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }The shard shard1 is consisted by three nodes: two mongod nodes and a arbiter node. Then, a new secondary nodes was added. Like below. shard1: , , , { "_id" : 3, "host" : "10.136.16.73:27032" } ] Now, how to add the member 3 node into the shard shard1? |
| Comments |
| Comment by William Zola [ 01/Aug/12 ] | |
|
Hi Jianfeng! Thanks for following-up. I'm having a hard time understanding your last response. 1) You're correct: the output of 'sh.status()' is the result of querying the data in the config server, and the 'config.shards' collection in particular. 2) You're correct: all three config servers must be up and running properly in order for there to be any changes to the metadata of the sharded cluster. If one of the config servers is down, then the metadata for the cluster becomes read-only. This means that if one of the config servers goes down, all chunk splitting and migration stops. This also means that if one of the config servers goes down, MongoDB cannot make changes to the "seed list" in the config server metadata. 3) You are also correct that you can continue to write data to the cluster, even if the metadata is read-only. This behavior is the result of deliberate design decisions by the MongoDB architects. You can read here for more information about those decisions: 4) It sounds like you're saying that your problem was due to one of your config servers being down. Is this the case? It also sounds like you're saying that bringing up the full set of config servers fixed your problem. Please let me know if my understanding is correct. Please let me know if you have further questions. -William | |
| Comment by Randolph Tan [ 01/Aug/12 ] | |
|
Hi, I have created a new ticket ( 1. Make sure all config servers are up and healthy. | |
| Comment by Jianfeng Xu [ 01/Aug/12 ] | |
|
Thanks! I guess that the mongos can listen a new node that is added into a mongod's replica set(refers log of mongos), but the config servers metadata are not synced to all of mongods and the collection "config.shards" data is error. So when using "sh.status()" to query, it just find the data of "config.shards". This case is easy to reproduct. It's a big problem when the config servers' metadata are not synced to all of mongods. For example, when a config server goes down, other of config servers is read-only, but the whole sharding is write-read(refers mongodb document) and APP write request is alway writing data to mongod. | |
| Comment by Randolph Tan [ 31/Jul/12 ] | |
|
There seems to be nothing wrong based on the mongos logs. Would it be possible to provide the verbose logs of all the config servers? You can set the logging to a higher verbosity without shutting down the server by using the setParam command:
and change it back later to the original value. | |
| Comment by Jianfeng Xu [ 31/Jul/12 ] | |
|
Hi, Randolph 1. 3 config servers. You can look at the previous detail comments i chat with William. Thanks | |
| Comment by Randolph Tan [ 30/Jul/12 ] | |
|
Hi, Couple of questions/requests: 1. How many config servers do you have? Thanks! | |
| Comment by Jianfeng Xu [ 30/Jul/12 ] | |
|
Hi William! Please look at the output of rs.status(), the "10.136.16.73:27032" node is in the SECONDARY status, but it's not work well and the sh.status() is not updated automatically. I also restarted all of mongos servers and config servers. I check all mongos logs and the mongos servers have successed to connect to "10.136.16.73:27032" node before they start. I guess that the config servers' metadata are not updated correctly. Any ideas? Thanks! ##rs.status()############################################################ , , , , shards: { "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" } d_name.t_name chunks: ###mongos log################################################################### from shard1/ from shard2/ | |
| Comment by William Zola [ 27/Jul/12 ] | |
|
Hi Jianfeng! All of that looks good. As I said before, the sh.status() should update automatically. Can you please send me the output of rs.status() from one of the nodes in shard1. It is possible that your new host is not yet in SECONDARY status, and the output of this command will check for that. Please let me know. Have a great day! -William | |
| Comment by Jianfeng Xu [ 27/Jul/12 ] | |
|
Thanks!
Fri Jul 27 09:09:39 [LockPinger] cluster 10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000 pinged successfully at Fri Jul 27 09:09:39 2012 by distributed lock pinger '10.136.20.57:20000,10.136.20.58:20000,10.136.20.59:20000/mongos-2057:30000:1343047408:1804289383', sleeping for 30000ms
[root@mongodb-1667 ~]# date shards: { "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" } d_name.t_name chunks: – popgo | |
| Comment by William Zola [ 26/Jul/12 ] | |
|
Hi Jianfeng! I'll need additional information to diagnose this problem. Please let me know the following:
I'd like to help you solve this problem. I can't move forward without this information. -William | |
| Comment by Jianfeng Xu [ 26/Jul/12 ] | |
|
Thanks for your reply. Yes, u can look the output of sh.status(). mongos> sh.status() shards: { "_id" : "shard1", "host" : "shard1/10.136.16.67:27032,10.136.16.68:27032" } { "_id" : "shard2", "host" : "shard2/10.136.16.69:27032,10.136.16.70:27032" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "d_name", "partitioned" : true, "primary" : "shard1" } d_name.t_name chunks: | |
| Comment by Randolph Tan [ 26/Jul/12 ] | |
|
It should be able to update itself over time. Is it not in your case? |