-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 6.0.12
-
Component/s: None
-
Labels:None
-
ALL
-
I have 5 node replicaset. where host-0 is the primary, host 1 to 4 are secondary.
I made the host-4 & host-4 down for some maintenance work.
The rs.status() was look like :
```
host-0 -> primary
host-1, host-2 -> secondary
host-3, host-4 -> not reachable/healthy
```
Then my rs-2 pod gets down for a connection issue, & after that I see there is no primary in the replicaset. Even the previous primary host-0 converts to secondary !!
I have no idea why.
Here are the member found from `rs.status()` output
members: [
{
_id: 0,
name: 'mg-rs-0.mg-rs-pods.demo.svc.cluster.local:27017',
health: 1,
state: 2,
stateStr: 'SECONDARY',
uptime: 1281,
optime: { ts: Timestamp(
), t: Long('3') },
optimeDate: ISODate('2024-04-08T12:58:21.000Z'),
lastAppliedWallTime: ISODate('2024-04-08T12:58:21.498Z'),
lastDurableWallTime: ISODate('2024-04-08T12:58:21.498Z'),
syncSourceHost: '',
syncSourceId: -1,
infoMessage: '',
configVersion: 9,
configTerm: 3,
self: true,
lastHeartbeatMessage: ''
},
{
_id: 1,
name: 'mg-rs-1.mg-rs-pods.demo.svc.cluster.local:27017',
health: 1,
state: 2,
stateStr: 'SECONDARY',
uptime: 1219,
optime: { ts: Timestamp(
), t: Long('3') },
optimeDurable: { ts: Timestamp(
), t: Long('3') },
optimeDate: ISODate('2024-04-08T12:58:21.000Z'),
optimeDurableDate: ISODate('2024-04-08T12:58:21.000Z'),
lastAppliedWallTime: ISODate('2024-04-08T12:58:21.498Z'),
lastDurableWallTime: ISODate('2024-04-08T12:58:21.498Z'),
lastHeartbeat: ISODate('2024-04-08T12:58:38.514Z'),
lastHeartbeatRecv: ISODate('2024-04-08T12:58:38.507Z'),
pingMs: Long('0'),
lastHeartbeatMessage: '',
syncSourceHost: '',
syncSourceId: -1,
infoMessage: '',
configVersion: 9,
configTerm: 3
},
{
_id: 2,
name: 'mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017',
health: 0,
state: 8,
stateStr: '(not reachable/healthy)',
uptime: 0,
optime: { ts: Timestamp(
), t: Long('-1') },
optimeDurable: { ts: Timestamp(
), t: Long('-1') },
optimeDate: ISODate('1970-01-01T00:00:00.000Z'),
optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'),
lastAppliedWallTime: ISODate('2024-04-08T12:58:04.600Z'),
lastDurableWallTime: ISODate('2024-04-08T12:58:04.600Z'),
lastHeartbeat: ISODate('2024-04-08T12:58:38.163Z'),
lastHeartbeatRecv: ISODate('2024-04-08T12:58:10.511Z'),
pingMs: Long('0'),
lastHeartbeatMessage: 'Error connecting to mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-2.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)',
syncSourceHost: '',
syncSourceId: -1,
infoMessage: '',
configVersion: 9,
configTerm: 3
},
{
_id: 3,
name: 'mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017',
health: 0,
state: 8,
stateStr: '(not reachable/healthy)',
uptime: 0,
optime: { ts: Timestamp(
), t: Long('-1') },
optimeDurable: { ts: Timestamp(
), t: Long('-1') },
optimeDate: ISODate('1970-01-01T00:00:00.000Z'),
optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'),
lastAppliedWallTime: ISODate('2024-04-08T12:45:44.788Z'),
lastDurableWallTime: ISODate('2024-04-08T12:45:44.788Z'),
lastHeartbeat: ISODate('2024-04-08T12:58:38.629Z'),
lastHeartbeatRecv: ISODate('2024-04-08T12:45:49.039Z'),
pingMs: Long('0'),
lastHeartbeatMessage: 'Error connecting to mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-3.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)',
syncSourceHost: '',
syncSourceId: -1,
infoMessage: '',
configVersion: 9,
configTerm: 1
},
{
_id: 4,
name: 'mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017',
health: 0,
state: 8,
stateStr: '(not reachable/healthy)',
uptime: 0,
optime: { ts: Timestamp(
), t: Long('-1') },
optimeDurable: { ts: Timestamp(
), t: Long('-1') },
optimeDate: ISODate('1970-01-01T00:00:00.000Z'),
optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'),
lastAppliedWallTime: ISODate('2024-04-08T12:45:24.787Z'),
lastDurableWallTime: ISODate('2024-04-08T12:45:24.787Z'),
lastHeartbeat: ISODate('2024-04-08T12:58:38.120Z'),
lastHeartbeatRecv: ISODate('2024-04-08T12:45:33.529Z'),
pingMs: Long('0'),
lastHeartbeatMessage: 'Error connecting to mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017 :: caused by :: Could not find address for mg-rs-4.mg-rs-pods.demo.svc.cluster.local:27017: SocketException: Host not found (authoritative)',
syncSourceHost: '',
syncSourceId: -1,
infoMessage: '',
configVersion: 9,
configTerm: 1
}