-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 4.4.5
-
Component/s: None
-
Labels:
-
Environment:Ubuntu 18.04.4 LTS
Problem Statement/Rationale
Sometimes our Mongodb replicaset has multiple primary nodes! in this case mongo-b node is our real primary and mongo-c node is not responding and just have primary role in fake.
Number of nodes in replicaset: 4 (3 normal + 1 delay node without voting). network partition and access checked and there is no partion. we have had this problem several times since updating the cluster from version 4.2.5 to version 4.4.5. each time we fix the problem by cleaning datadir on fake node and restarting it.
Steps to Reproduce
it happens time to time. it doesn't have simple approach to Reproduce
Expected Results
cluster should have only one primary or produce better log about issue.
Actual Results
// { [1881/1930] "_id" : 5, "name" : "mongo-c:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 3435, "optime" : { "ts" : Timestamp(1637525581, 2), "t" : NumberLong(87) }, "optimeDurable" : { "ts" : Timestamp(1637525443, 1), "t" : NumberLong(87) }, "optimeDate" : ISODate("2021-11-21T20:13:01Z"), "optimeDurableDate" : ISODate("2021-11-21T20:10:43Z"), "lastHeartbeat" : ISODate("2021-11-24T07:08:32.108Z"), "lastHeartbeatRecv" : ISODate("2021-11-24T07:08:34.867Z"), "pingMs" : NumberLong(2355), "lastHeartbeatMessage" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1637524307, 1), "electionDate" : ISODate("2021-11-21T19:51:47Z"), "configVersion" : 23, "configTerm" : 87 }, { "_id" : 6, "name" : "mongo-b:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 7995003, "optime" : { "ts" : Timestamp(1637737714, 24), "t" : NumberLong(88) }, "optimeDate" : ISODate("2021-11-24T07:08:34Z"), "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1637525581, 1), "electionDate" : ISODate("2021-11-21T20:13:01Z"), "configVersion" : 23, "configTerm" : 88, "self" : true, "lastHeartbeatMessage" : "" }
Additional Notes
replica conf:
{ [25/599]
"_id" : 0,
"host" : "mongo-a:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "mongo-h:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : true,
"priority" : 0,
"tags" : {
},
"slaveDelay" : NumberLong(900),
"votes" : 0
},
{
"_id" : 5,
"host" : "mongo-c:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 6,
"host" : "mongo-b:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}