[SERVER-56133] Why lastHeartbeatRecv don not updated Created: 16/Apr/21  Updated: 01/Jun/22  Resolved: 22/Apr/21

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.4.14, 4.0.12
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: zhijia zhang Assignee: Edwin Zhou
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Hi all i met a strange question .

Mongodb version 4.0.12 community

An replset with node A(primary node),B(sec node),C(sec node) i want to copy node B to new machine and then remove it .

My detail operation is login the pri and set the node B priority is 0 and hidden is true then reconfig it . i see it work and node B transfer to hidden node , next login the node B and exec db.fsyncLock() , it also work , next i scp node B dir to new machine then start it success and add it in the replset .

so as for now in the replset i have 4 nodes , ABCD ,but in the pri i exec rs.status() i see node B lastHeartbeatRecv still was t1(the time i add node C in the replset), and it won't changed . but lastHeartbeat is still change to the last.

And if i exec rs.status() in the node B i just can see node A and node C i know it cause of node B was locked and can not update config , but in the result the node A and node C lastHeartbeat and lastHeartbeatRecv still was t1.

I exec the db.setLogLevel(2,"replication.heartbeats"); in the pri node and i can see in the mongod.log the node B heartbeat request to node A .

So why lastHeartbeatRecv is not updated in the node A and node C and node D ?



 Comments   
Comment by Edwin Zhou [ 22/Apr/21 ]

Hi aaa15942975658@163.com,

It appears that after running db.fsyncLock() on secondary B then adding a node to a replica set will cause this issue to occur. You will need to run db.fsyncUnlock() on the secondary B node.

Since you are adding and removing nodes from a replica set, I recommend reviewing the documentation for adding members to a replica set and removing nodes from a replica set

Just as a reminder, the SERVER project is for bugs and feature suggestions for the MongoDB server. As this ticket does not appear to be a bug, I will now close it. If you need further assistance troubleshooting, I encourage you to ask our community by posting on the MongoDB Developer Community Forums.

Best,
Edwin

Comment by zhijia zhang [ 19/Apr/21 ]

the detail operation like this :
in the primary node choose a sec node (NODEB)set the priority 0 and hidden true and reconfig it .

a=rs.config()
a.members[1].priority=0
a.members[1].hidden=true
rs.reconfig(a)

then in the NODEB that was hidden exec
db.fsyncLock()

you can see whatever in the pri node and NODEB that config version was same and lastHeartbeat and lastHeartbeatRecv was still updated.

next you can scp -r NODEB dir to the new machine and then start it and add it to the replset . TIME1

after that you can exec rs.status in the pri node you will see that NODEB lastHeartbeat is still updated but lastHeartbeatRecv was TIME1 and will not change anymore.
also you can exec rs.status in the NODEB and you will see pri node and sec node's lastHeartbeat and lastHeartbeatRecv was TIME1 and also not change anymore.

i exec db.setLogLevel(2,"replication.heartbeats"); in the pri node and sec node .
in the node log i can see NODEB send request to the pri node and also pri node will send response to the NODEB ,
so why lastHeartbeatRecv was not changed in the pri and other sec node ? and why in the NODEB pri and other sec lastHeartbeat and lastHeartbeatRecv was not changed

Generated at Thu Feb 08 05:38:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.