[SERVER-34816] Primary Mongo Replica set crashed in Production env Created: 02/May/18 Updated: 08/May/18 Resolved: 02/May/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Sai Kiran Meduri | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
We have one primary and two secondary replica sets for our production env and the primary replica set is crashed every minute when I bring it to PRIMARY state and when I check the status of Mongod service it shows as failed: mongod.service - SYSV: Mongo is a scalable, document-oriented database. Loaded: loaded (/etc/rc.d/init.d/mongod) Active: failed (Result: exit-code) since Wed 2018-05-02 21:56:03 UTC; 9min ago Process: 27285 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE) production mongod[27285]: Error starting mongod. /var/run/mongodb/mongod.pid exists. production systemd[1]: mongod.service: control process exited, code=exited status=1 production systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database.. production systemd[1]: Unit mongod.service entered failed state.
Tried restarting the service several times and still unable to get the Primary replica set to primary state. As followed from other blogs, I've tried deleting the /tmp/mongodb-27017.sock file several times and restarted the service with our .congf file. I have the Logs generated in /var/log/mongodb/mongod.log *2017-11-25T09:01:33.834+0000 I CONTROL [main] ***** SERVER RESTARTED ****** 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] MongoDB starting : pid=16025 port=27017 dbpath=/var/lib/mongo 64-bit host=new-mittens-production 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] db version v3.2.12 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] modules: none 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] build environment: 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] distmod: rhel70 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] distarch: x86_64 2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] target_arch: x86_64 *2017-11-25T09:01:33.846+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true }}, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }* 2017-11-25T09:01:33.877+0000 E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted 2017-11-25T09:01:33.877+0000 I - [initandlisten] Fatal Assertion 28578 2017-11-25T09:01:33.877+0000 I - [initandlisten] ** ***aborting after fassert() failure
Any suggestions or help is appreciated. |
| Comments |
| Comment by Kelsey Schubert [ 02/May/18 ] |
|
Hi saimeduri, Thanks for your report. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-users group. I'd also suggest double-checking your permissions. Kind regards, |