[SERVER-34816] Primary Mongo Replica set crashed in Production env Created: 02/May/18  Updated: 08/May/18  Resolved: 02/May/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sai Kiran Meduri Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

We have one primary and two secondary replica sets for our production env and the primary replica set is crashed every minute when I bring it to PRIMARY state and when I check the status of Mongod service it shows as failed: 

mongod.service - SYSV: Mongo is a scalable, document-oriented database.

   Loaded: loaded (/etc/rc.d/init.d/mongod)

   Active: failed (Result: exit-code) since Wed 2018-05-02 21:56:03 UTC; 9min ago

  Process: 27285 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

production mongod[27285]: Error starting mongod. /var/run/mongodb/mongod.pid exists.

production systemd[1]: mongod.service: control process exited, code=exited status=1

production systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..

production systemd[1]: Unit mongod.service entered failed state.

 

Tried restarting the service several times and still unable to get the Primary replica set to primary state. As followed from other blogs, I've tried deleting the /tmp/mongodb-27017.sock file several times and restarted the service with our .congf file.

I have the Logs generated in /var/log/mongodb/mongod.log 

*2017-11-25T09:01:33.834+0000 I CONTROL  [main] ***** SERVER RESTARTED ******

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] MongoDB starting : pid=16025 port=27017 dbpath=/var/lib/mongo 64-bit host=new-mittens-production

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] db version v3.2.12

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] allocator: tcmalloc

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] modules: none

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] build environment:

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     distmod: rhel70

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     distarch: x86_64

2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     target_arch: x86_64

*2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net:

{ bindIp: "0.0.0.0", port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }*

2017-11-25T09:01:33.877+0000 E NETWORK  [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted

2017-11-25T09:01:33.877+0000 I -        [initandlisten] Fatal Assertion 28578

2017-11-25T09:01:33.877+0000 I -        [initandlisten]

 ** 

***aborting after fassert() failure

 

Any suggestions or help is appreciated. 



 Comments   
Comment by Kelsey Schubert [ 02/May/18 ]

Hi saimeduri,

Thanks for your report. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group.

I'd also suggest double-checking your permissions.

Kind regards,
Kelsey

Generated at Thu Feb 08 04:37:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.