[SERVER-4453] Shards not picking up replica set updates Created: 08/Dec/11  Updated: 11/Jul/16  Resolved: 02/Jan/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dmytro Korsakov Assignee: Greg Studer
Resolution: Done Votes: 0
Labels: mongos, replicaset, replication, sharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 10.04 on Amazon EC2


Issue Links:
Duplicate
duplicates SERVER-4399 mongos doesn't always update shards i... Closed
Related
Operating System: Linux
Participants:

 Description   

When adding next replica to set we expect mongo to add this node to shard automatically, but in our setup it has never happened.

e.g. after successfully adding the second replica to set we get rs config:

PRIMARY> rs.status()
{
"set" : "rs-0",
"date" : ISODate("2011-12-08T15:46:26Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "mongo-0-0:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"optime" :

{ "t" : 1323358847000, "i" : 1 }

,
"optimeDate" : ISODate("2011-12-08T15:40:47Z"),
"self" : true
},
{
"_id" : 1,
"name" : "mongo-0-1:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 513,
"optime" :

{ "t" : 1323358847000, "i" : 1 }

,
"optimeDate" : ISODate("2011-12-08T15:40:47Z"),
"lastHeartbeat" : ISODate("2011-12-08T15:46:26Z"),
"pingMs" : 2
},
{
"_id" : 2,
"name" : "mongo-0-0:27020",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 327,
"optime" :

{ "t" : 0, "i" : 0 }

,
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2011-12-08T15:46:25Z"),
"pingMs" : 0
}
],
"ok" : 1
}

Then we go on router and flush router's config to update cached info:

mongos> db.runCommand("flushRouterConfig")

{ "flushed" : true, "ok" : 1 }

And the new node 'mongo-0-1' is not in shard-0:

mongos> db.printShardingStatus()
— Sharding Status —
sharding version:

{ "_id" : 1, "version" : 3 }

shards:

{ "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }

databases:

{ "_id" : "admin", "partitioned" : false, "primary" : "config" }

Then we check config-server, just to make sure that shard knows nothing about any new changes in replica set:

db.shards.find()

{ "_id" : "shard-0", "host" : "rs-0/mongo-0-0:27018" } { "_id" : "shard-1", "host" : "rs-1/mongo-1-0:27018" }

The only suspicious part in router's log was:

Thu Dec 8 15:37:47 [ReplicaSetMonitorWatcher] updated set (rs-0) to: rs-0/mongo-0-0:27018,mongo-0-1:27018
Thu Dec 8 15:37:47 [ReplicaSetMonitorWatcher] warning: node: mongo-0-1:27018 isn't a part of set: rs-0 ismaster:

{ ismaster: false, secondary: false, info: "can't get local.system.replset config from self or any seed (EMPTYCONFIG)", isreplicaset: true, maxBsonObjectSize: 16777216, ok: 1.0 }

For now we decided to update shards collection manually on each change in replica set, but it would be great to delegate this job to mongo just as official documentation states.



 Comments   
Comment by Daniel Pasette (Inactive) [ 06/Mar/12 ]

this was backported to 2.0.3. Information and commit history in SERVER-4399.

Comment by Greg Studer [ 02/Jan/12 ]

Fixed in 2.1 branch, mongos now updates hosts as it detects them changing.

Comment by Greg Studer [ 08/Dec/11 ]

Assuming that message eventually stopped in the ReplicaSetMonitorWatcher (and the watcher reports the full replica set if/when you set the loglevel up to 2), you should watch SERVER-4399 for better updating of replica set state.

Comment by Dmytro Korsakov [ 08/Dec/11 ]

All hostnames were double-checked and since the router process is placed on the same physical node with primary, the secondary, which has been successfully added to the set, cannot have any hostname related issues

Comment by Eliot Horowitz (Inactive) [ 08/Dec/11 ]

Can you check that the hostnames are resolving to the same place?

Generated at Thu Feb 08 03:06:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.