[SERVER-56299] Mongodb-Replica set Crashes Unexpectedly with version 4.0.23 Created: 23/Apr/21  Updated: 19/May/21  Resolved: 19/May/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.23
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Rakhi Maheshwari Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: performance, stability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Hi,

We have a three node replica set cluster with one primary and two secondary nodes, due to slow performance and high query execution time on version 3.6, we have upgraded mongodb version to 4.0.23, but after upgrade when application load is started again then mongodb primary server crashes unexpectedly with high CPU usage to 80% and becomes SECONDARY with mongo02 reaching into unhealthy state.

Also the states of replicaset changes continuously over the time as:

SECONDARY, UNHEALTHY, SECONDARY

SECONDARY, SECONDARY,SECONDARY

SECONDARY, PRIMARY PRIMARY.

Please find one of the state here:

"members" : [ "members" : [ { "_id" : 0, "name" : "PRIMARY:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" :

{ "ts" : Timestamp(1618811595, 1373), "t" : NumberLong(106) }

, "optimeDate" : ISODate("2021-04-19T05:53:15Z"), "syncingTo" : "SECONDARY02:27017", "syncSourceHost" : "SECONDARY02:27017", "syncSourceId" : 4, "infoMessage" : "", "configVersion" : 19, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "SECONDARY01:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 186, "optime" :

{ "ts" : Timestamp(1618818925, 1), "t" : NumberLong(141) }

, "optimeDurable" : { "ts" : Timestamp(1618818925, 1), "t" : NumberLong(141) }, "optimeDate" : ISODate("2021-04-19T07:55:25Z"), "optimeDurableDate" : ISODate("2021-04-19T07:55:25Z"), "lastHeartbeat" : ISODate("2021-04-19T07:55:32.430Z"), "lastHeartbeatRecv" : ISODate("2021-04-19T07:55:30.739Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1618818922, 1), "electionDate" : ISODate("2021-04-19T07:55:22Z"), "configVersion" : 19 }, { "_id" : 4, "name" : "SECONDARY02:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 186, "optime" :

{ "ts" : Timestamp(1618813475, 1405), "t" : NumberLong(106) }

, "optimeDurable" : { "ts" : Timestamp(1618813475, 1405), "t" : NumberLong(106) }, "optimeDate" : ISODate("2021-04-19T06:24:35Z"), "optimeDurableDate" : ISODate("2021-04-19T06:24:35Z"), "lastHeartbeat" : ISODate("2021-04-19T07:55:32.428Z"), "lastHeartbeatRecv" : ISODate("2021-04-19T07:55:32.260Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1618818892, 2), "electionDate" : ISODate("2021-04-19T07:54:52Z"), "configVersion" : 19 }, ],

 

 



 Comments   
Comment by Dmitry Agranat [ 19/May/21 ]

Hi rakhi.m@vvdntech.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 11/May/21 ]

Hi rakhi.m@vvdntech.com,

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information?

Thanks,
Dima

Comment by Dmitry Agranat [ 26/Apr/21 ]

Hi rakhi.m@vvdntech.com,

For each member of the replica set, please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Please mention the exact start/stop timestamp and timezone of the event.

Dima

Generated at Thu Feb 08 05:38:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.