Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10722

MongoDB periodically segfaults, no error messages in mongo logs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      Linux

      Description

      My mongodb process dies arbitrarily without any logs. It has happened a couple of times over the last few weeks. I had to manually start the mongod server. I looked at the system logs and it was not an OOM kill.

      I am on AWS, Amzon linux XLarge instance.

      Here are the mongod logs from when it died. I would like to understand if I can turn on some logging to determine the root cause if it happens again.

      Mon Sep  9 10:38:28.554 [conn895640] end connection 10.139.25.254:42073 (246 connections now open)
      Mon Sep  9 10:38:28.554 [conn895642] end connection 10.139.25.254:42074 (248 connections now open)
      Mon Sep  9 10:38:28.594 [conn895644] end connection 10.139.25.254:42076 (242 connections now open)
      Mon Sep  9 10:38:28.594 [conn895648] end connection 10.139.25.254:42080 (241 connections now open)
      Mon Sep  9 10:38:28.594 [conn895652] end connection 10.139.25.254:42084 (240 connections now open)
      Mon Sep  9 10:38:28.595 [conn895646] end connection 10.139.25.254:42079 (239 connections now open)
      Mon Sep  9 10:38:28.595 [conn895650] end connection 10.139.25.254:42082 (239 connections now open)
      Mon Sep  9 10:38:28.595 [conn895653] end connection 10.139.25.254:42086 (237 connections now open)
      Mon Sep  9 10:38:28.595 [initandlisten] connection accepted from 10.139.25.254:42110 #895718 (239 connections now open)
      Mon Sep  9 10:38:28.595 [initandlisten] connection accepted from 10.139.25.254:42109 #895719 (239 connections now open)
      Mon Sep  9 10:38:28.596 [conn895647] end connection 10.139.25.254:42078 (238 connections now open)
      Mon Sep  9 10:38:28.596 [conn895651] end connection 10.139.25.254:42083 (238 connections now open)
      Mon Sep  9 10:38:28.596 [conn895654] end connection 10.139.25.254:42087 (236 connections now open)
      Mon Sep  9 10:38:28.596 [conn895649] end connection 10.139.25.254:42081 (235 connections now open)
      Mon Sep  9 10:38:28.596 [conn895645] end connection 10.139.25.254:42077 (235 connections now open)
      Mon Sep  9 10:38:28.596 [conn895655] end connection 10.139.25.254:42085 (233 connections now open)
      Mon Sep  9 10:38:28.601 [initandlisten] connection accepted from 10.139.25.254:42111 #895720 (234 connections now open)
      Mon Sep  9 10:38:28.601 [initandlisten] connection accepted from 10.139.25.254:42112 #895721 (235 connections now open)
      Mon Sep  9 10:38:28.603 [initandlisten] connection accepted from 10.139.25.254:42113 #895722 (236 connections now open)
      Mon Sep  9 10:38:28.604 [initandlisten] connection accepted from 10.139.25.254:42114 #895723 (237 connections now open)
       
       
      ***** SERVER RESTARTED *****
       
       
      Mon Sep  9 14:12:11.108 [initandlisten] MongoDB starting : pid=29285 port=27017 dbpath=/mongodb_data 64-bit host=ip-10-166-29-94
      Mon Sep  9 14:12:11.109 [initandlisten] db version v2.4.1
      Mon Sep  9 14:12:11.109 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110
      Mon Sep  9 14:12:11.109 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
      Mon Sep  9 14:12:11.109 [initandlisten] allocator: tcmalloc
      Mon Sep  9 14:12:11.109 [initandlisten] options: { auth: "true", config: "/etc/mongod.conf", dbpath: "/mongodb_data", fork: "true", keyFile: "/var/lib/mongo/rskey", logappend: "true", logpat
      h: "/var/log/mongo/mongod.log", pidfilepath: "/var/run/mongodb/mongod.pid", replSet: "RS-aviarymongo-571" }
      Mon Sep  9 14:12:11.117 [initandlisten] journal dir=/mongodb_data/journal
      Mon Sep  9 14:12:11.117 [initandlisten] recover begin
      Mon Sep  9 14:12:11.117 [initandlisten] recover lsn: 299400579
      Mon Sep  9 14:12:11.117 [initandlisten] recover /mongodb_data/journal/j._3
      Mon Sep  9 14:12:11.132 [initandlisten] recover cleaning up
      Mon Sep  9 14:12:11.132 [initandlisten] removeJournalFiles
      Mon Sep  9 14:12:11.310 [initandlisten] recover done
      Mon Sep  9 14:12:11.310 [initandlisten] preallocating a journal file /mongodb_data/journal/prealloc.0
      Mon Sep  9 14:12:40.763 [websvr] admin web console waiting for connections on port 28017
      Mon Sep  9 14:12:40.763 [initandlisten] waiting for connections on port 27017

        Attachments

          Activity

            People

            Assignee:
            samantha.ritter Samantha Ritter (Inactive)
            Reporter:
            dharshanr@scalegrid.net Dharshan Rangegowda
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: