-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Replication
-
ALL
-
(copied to CRM)
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Currently a secondary will postpone its election timeout if we receive a heartbeat response from the primary node. This means if the primary is in a bad state and is unable to send heartbeat requests to secondaries, but it can respond to heartbeats, it will never step down.
This can lead to a scenario where a mongod that cannot do any work (due to a host issue leading to CPU / scheduling issues) is still alive and stays primary, when a failover would bring the replica set back to being healthy and accepting incoming connections.
One potential solution is to have secondaries also check if they have received a heartbeat request from the primary within their election timeout
The consequences of doing nothing here is leaving open a scenario where we lose availability on the cluster until there is manual intervention done via a forced stepup / failover on the cluster