[SERVER-4453] Shards not picking up replica set updates Created: 08/Dec/11 Updated: 11/Jul/16 Resolved: 02/Jan/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 2.0.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Dmytro Korsakov | Assignee: | Greg Studer |
| Resolution: | Done | Votes: | 0 |
| Labels: | mongos, replicaset, replication, sharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 10.04 on Amazon EC2 |
||
| Issue Links: |
|
||||||||||||
| Operating System: | Linux | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
When adding next replica to set we expect mongo to add this node to shard automatically, but in our setup it has never happened. e.g. after successfully adding the second replica to set we get rs config: PRIMARY> rs.status() , , , Then we go on router and flush router's config to update cached info: mongos> db.runCommand("flushRouterConfig") { "flushed" : true, "ok" : 1 }And the new node 'mongo-0-1' is not in shard-0: mongos> db.printShardingStatus() shards: { "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" }Then we check config-server, just to make sure that shard knows nothing about any new changes in replica set: db.shards.find() { "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }The only suspicious part in router's log was: Thu Dec 8 15:37:47 [ReplicaSetMonitorWatcher] updated set (rs-0) to: rs-0/mongo-0-0:27018,mongo-0-1:27018
For now we decided to update shards collection manually on each change in replica set, but it would be great to delegate this job to mongo just as official documentation states. |
| Comments |
| Comment by Daniel Pasette (Inactive) [ 06/Mar/12 ] |
|
this was backported to 2.0.3. Information and commit history in |
| Comment by Greg Studer [ 02/Jan/12 ] |
|
Fixed in 2.1 branch, mongos now updates hosts as it detects them changing. |
| Comment by Greg Studer [ 08/Dec/11 ] |
|
Assuming that message eventually stopped in the ReplicaSetMonitorWatcher (and the watcher reports the full replica set if/when you set the loglevel up to 2), you should watch |
| Comment by Dmytro Korsakov [ 08/Dec/11 ] |
|
All hostnames were double-checked and since the router process is placed on the same physical node with primary, the secondary, which has been successfully added to the set, cannot have any hostname related issues |
| Comment by Eliot Horowitz (Inactive) [ 08/Dec/11 ] |
|
Can you check that the hostnames are resolving to the same place? |