Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34816

Primary Mongo Replica set crashed in Production env

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • ALL

    Description

      We have one primary and two secondary replica sets for our production env and the primary replica set is crashed every minute when I bring it to PRIMARY state and when I check the status of Mongod service it shows as failed: 

      mongod.service - SYSV: Mongo is a scalable, document-oriented database.

         Loaded: loaded (/etc/rc.d/init.d/mongod)

         Active: failed (Result: exit-code) since Wed 2018-05-02 21:56:03 UTC; 9min ago

        Process: 27285 ExecStart=/etc/rc.d/init.d/mongod start (code=exited, status=1/FAILURE)

      production mongod[27285]: Error starting mongod. /var/run/mongodb/mongod.pid exists.

      production systemd[1]: mongod.service: control process exited, code=exited status=1

      production systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..

      production systemd[1]: Unit mongod.service entered failed state.

       

      Tried restarting the service several times and still unable to get the Primary replica set to primary state. As followed from other blogs, I've tried deleting the /tmp/mongodb-27017.sock file several times and restarted the service with our .congf file.

      I have the Logs generated in /var/log/mongodb/mongod.log 

      *2017-11-25T09:01:33.834+0000 I CONTROL  [main] ***** SERVER RESTARTED ******

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] MongoDB starting : pid=16025 port=27017 dbpath=/var/lib/mongo 64-bit host=new-mittens-production

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] db version v3.2.12

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] git version: ef3e1bc78e997f0d9f22f45aeb1d8e3b6ac14a14

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] allocator: tcmalloc

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] modules: none

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] build environment:

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     distmod: rhel70

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     distarch: x86_64

      2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten]     target_arch: x86_64

      *2017-11-25T09:01:33.846+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net:

      { bindIp: "0.0.0.0", port: 27017 }

      , processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal:

      { enabled: true }

      }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }*

      2017-11-25T09:01:33.877+0000 E NETWORK  [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted

      2017-11-25T09:01:33.877+0000 I -        [initandlisten] Fatal Assertion 28578

      2017-11-25T09:01:33.877+0000 I -        [initandlisten]

       ** 

      ***aborting after fassert() failure

       

      Any suggestions or help is appreciated. 

      Attachments

        Activity

          People

            kelsey.schubert@mongodb.com Kelsey Schubert
            saimeduri Sai Kiran Meduri
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: