[SERVER-56299] Mongodb-Replica set Crashes Unexpectedly with version 4.0.23 Created: 23/Apr/21 Updated: 19/May/21 Resolved: 19/May/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.0.23 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Rakhi Maheshwari | Assignee: | Dmitry Agranat |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | performance, stability | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Participants: |
| Description |
|
Hi, We have a three node replica set cluster with one primary and two secondary nodes, due to slow performance and high query execution time on version 3.6, we have upgraded mongodb version to 4.0.23, but after upgrade when application load is started again then mongodb primary server crashes unexpectedly with high CPU usage to 80% and becomes SECONDARY with mongo02 reaching into unhealthy state. Also the states of replicaset changes continuously over the time as: SECONDARY, UNHEALTHY, SECONDARY SECONDARY, SECONDARY,SECONDARY SECONDARY, PRIMARY PRIMARY. Please find one of the state here: "members" : [ "members" : [ { "_id" : 0, "name" : "PRIMARY:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 244, "optime" : { "ts" : Timestamp(1618811595, 1373), "t" : NumberLong(106) }, "optimeDate" : ISODate("2021-04-19T05:53:15Z"), "syncingTo" : "SECONDARY02:27017", "syncSourceHost" : "SECONDARY02:27017", "syncSourceId" : 4, "infoMessage" : "", "configVersion" : 19, "self" : true, "lastHeartbeatMessage" : "" }, { "_id" : 1, "name" : "SECONDARY01:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 186, "optime" : { "ts" : Timestamp(1618818925, 1), "t" : NumberLong(141) }, "optimeDurable" : { "ts" : Timestamp(1618818925, 1), "t" : NumberLong(141) }, "optimeDate" : ISODate("2021-04-19T07:55:25Z"), "optimeDurableDate" : ISODate("2021-04-19T07:55:25Z"), "lastHeartbeat" : ISODate("2021-04-19T07:55:32.430Z"), "lastHeartbeatRecv" : ISODate("2021-04-19T07:55:30.739Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1618818922, 1), "electionDate" : ISODate("2021-04-19T07:55:22Z"), "configVersion" : 19 }, { "_id" : 4, "name" : "SECONDARY02:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 186, "optime" : { "ts" : Timestamp(1618813475, 1405), "t" : NumberLong(106) }, "optimeDurable" : { "ts" : Timestamp(1618813475, 1405), "t" : NumberLong(106) }, "optimeDate" : ISODate("2021-04-19T06:24:35Z"), "optimeDurableDate" : ISODate("2021-04-19T06:24:35Z"), "lastHeartbeat" : ISODate("2021-04-19T07:55:32.428Z"), "lastHeartbeatRecv" : ISODate("2021-04-19T07:55:32.260Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1618818892, 2), "electionDate" : ISODate("2021-04-19T07:54:52Z"), "configVersion" : 19 }, ],
|
| Comments |
| Comment by Dmitry Agranat [ 19/May/21 ] |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. Regards, |
| Comment by Dmitry Agranat [ 11/May/21 ] |
|
Hi, We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information? Thanks, |
| Comment by Dmitry Agranat [ 26/Apr/21 ] |
|
For each member of the replica set, please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Please mention the exact start/stop timestamp and timezone of the event. Dima |