[SERVER-13045] issue with cluster Created: 05/Mar/14  Updated: 10/Dec/14  Resolved: 05/Mar/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Dwayne Mcnab Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

I have a three sharded cluster running in a test environment. Problem is when I try to query the dbs am getting an error message that the other two shards does not have a primary.

ar 5 05:34:19.872 listDatabases failed:{
"code" : 10009,
"ok" : 0,
"errmsg" : "exception: ReplicaSetMonitor no master found for set: SHARD2"
} at src/mongo/shell/mongo.js:46

Now when I log into the other two shards I am seeing that they are both logged in as secondary. The first that Shard that i have created is showing up as primary. Any idea why this would happen?

Also, I have given one of my db's dbAdmin and readWriteAdmin on any databases, can you tell me why he would be unable to query the dbs? He ran show dbs from the admin db and got an unauthorized error message.

Please advise. We are beginning to run heavy tests, We are almost ready to start running mongo in our production environment as we have already started converting our application from mysql.

thanks.



 Comments   
Comment by Dwayne Mcnab [ 06/Mar/14 ]

Ok Andy,

One of my DBA's had inserted by mistake some document into the configserver, could that have caused the problem? What i did to resolve this was to restart the mongos service. Then I noticed that I was able to synchronize the databases.

Thanks again.

Comment by Andy Schwerin [ 05/Mar/14 ]

Dwayne, I'm going to resolve this ticket. Please re-open it if you see the situation again, or if you have a set of reproduction steps.

Comment by Andy Schwerin [ 05/Mar/14 ]

The current state of the replicasets look healthy – each has one primary node, they're getting what look like regular heartbeats from one another. If the problem happens again, please capture the rs.status() right away, and we'll try to catch it in the act.

Comment by Dwayne Mcnab [ 05/Mar/14 ]

I am running v2.4.9.

{
"set" : "SHARD1",
"date" : ISODate("2014-03-05T14:58:01Z"),
"myState" : 1,
"members" : [

{ "_id" : 0, "name" : "mongo-27-1:27011", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 96750, "optime" : Timestamp(1393427829, 1), "optimeDate" : ISODate("2014-02-26T15:17:09Z"), "self" : true }

,

{ "_id" : 1, "name" : "mongo-27-4:27041", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 62950, "optime" : Timestamp(1393427829, 1), "optimeDate" : ISODate("2014-02-26T15:17:09Z"), "lastHeartbeat" : ISODate("2014-03-05T14:58:01Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T14:58:00Z"), "pingMs" : 0, "syncingTo" : "mongo-27-1:27011" }

,

{ "_id" : 2, "name" : "mongo-27-5:27051", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 62946, "optime" : Timestamp(1393427829, 1), "optimeDate" : ISODate("2014-02-26T15:17:09Z"), "lastHeartbeat" : ISODate("2014-03-05T14:58:00Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T14:58:00Z"), "pingMs" : 0, "syncingTo" : "mongo-27-1:27011" }

],
"ok" : 1
}
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{
"set" : "SHARD2",
"date" : ISODate("2014-03-05T15:00:07Z"),
"myState" : 2,
"syncingTo" : "mongo-27-6:27062",
"members" : [

{ "_id" : 0, "name" : "mongo-27-2:27022", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 538368, "optime" : Timestamp(1393493771, 1), "optimeDate" : ISODate("2014-02-27T09:36:11Z"), "self" : true }

,

{ "_id" : 1, "name" : "mongo-27-5:27052", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 23776, "optime" : Timestamp(1393493771, 1), "optimeDate" : ISODate("2014-02-27T09:36:11Z"), "lastHeartbeat" : ISODate("2014-03-05T15:00:06Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T15:00:05Z"), "pingMs" : 0, "syncingTo" : "mongo-27-6:27062" }

,

{ "_id" : 2, "name" : "mongo-27-6:27062", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 23778, "optime" : Timestamp(1393493771, 1), "optimeDate" : ISODate("2014-02-27T09:36:11Z"), "lastHeartbeat" : ISODate("2014-03-05T15:00:07Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T15:00:07Z"), "pingMs" : 0, "syncingTo" : "mongo-27-2:27022" }

],
"ok" : 1
}
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
rs.status()
{
"set" : "SHARD3",
"date" : ISODate("2014-03-05T12:01:46Z"),
"myState" : 2,
"syncingTo" : "mongo-27-6:27063",
"members" : [

{ "_id" : 0, "name" : "mongo-27-3:27033", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 605924, "optime" : Timestamp(1393485662, 1), "optimeDate" : ISODate("2014-02-27T07:21:02Z"), "self" : true }

,

{ "_id" : 1, "name" : "mongo-27-4:27043", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 22442, "optime" : Timestamp(1393485662, 1), "optimeDate" : ISODate("2014-02-27T07:21:02Z"), "lastHeartbeat" : ISODate("2014-03-05T12:01:45Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T12:01:46Z"), "pingMs" : 0, "syncingTo" : "mongo-27-6:27063" }

,

{ "_id" : 2, "name" : "mongo-27-6:27063", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 22445, "optime" : Timestamp(1393485662, 1), "optimeDate" : ISODate("2014-02-27T07:21:02Z"), "lastHeartbeat" : ISODate("2014-03-05T12:01:45Z"), "lastHeartbeatRecv" : ISODate("2014-03-05T12:01:45Z"), "pingMs" : 0, "syncingTo" : "mongo-27-3:27033" }

],
"ok" : 1
}

Thanks again Andy,

Comment by Andy Schwerin [ 05/Mar/14 ]

I think we've miscommunicated. I meant what version of the MongoDB binaries are you running on the shards. You can get the answer by running mongod --version, or looking at the top of the logs on start-up. I'm pretty sure you're running some version of 2.4 (maybe 2.4.9?) from your original problem with listDatabsaes. However, I don't think the precise version is needed to make progress.

Please run rs.status() in the shell against each of the misbehaving shards, and post the results, please.

Comment by Dwayne Mcnab [ 05/Mar/14 ]

First question, answer:
Shard nodes are running version 3.

Second question, answer:
each replica set has 3 members, two secondary and one primary.

Thanks for clearing up the clouds around access control. That does explains why dbAdminAnyDatabase wont list databases in version 2.6. However, I am still curious as to why two of my shards are showing up without primaries.

Thanks again Andy.

Comment by Andy Schwerin [ 05/Mar/14 ]

What version of MongoDB is running on your shard nodes? Also, could you describe your topology in a little more detail? You have a sharded cluster with three shards, each of which is a replica set. How many members does each replica set have?

As to your other question, about access control, if you're running MongoDB 2.4, your user will require the "clusterAdmin" role to list databases. In 2.6, any of the "any database" built-in roles have the required privileges to list databases.

Generated at Thu Feb 08 03:30:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.