Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20342

Corruption, or an unclean shutdown while writing the first section in a journal file

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Cannot Reproduce
    • Icon: Major - P3 Major - P3
    • None
    • 2.6.5
    • Storage
    • None

    Description

      We are currently deploying MongoDB on a number of sites, running Ubuntu 12.04 precise. On one of these systems, after running MongoDB for a few weeks, we discovered that the 'mongod' service had stopped. Trying to start it gave the following log:

      2015-09-01T15:22:23.447+0200 ***** SERVER RESTARTED *****
      2015-09-01T15:22:23.463+0200 [initandlisten] MongoDB starting : pid=31061 port=27017 dbpath=/var/lib/mongodb 64-bit host=intelecom-edda-ferd-gondan-pcie-cb1
      2015-09-01T15:22:23.463+0200 [initandlisten] db version v2.6.5
      2015-09-01T15:22:23.463+0200 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
      2015-09-01T15:22:23.463+0200 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
      2015-09-01T15:22:23.463+0200 [initandlisten] allocator: tcmalloc
      2015-09-01T15:22:23.463+0200 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
      2015-09-01T15:22:23.464+0200 [initandlisten] journal dir=/var/lib/mongodb/journal
      2015-09-01T15:22:23.464+0200 [initandlisten] recover begin
      2015-09-01T15:22:23.471+0200 [initandlisten] recover lsn: 2811296823
      2015-09-01T15:22:23.471+0200 [initandlisten] recover /var/lib/mongodb/journal/j._12
      2015-09-01T15:22:23.479+0200 [initandlisten] Journal file header invalid. This could indicate corruption, or an unclean shutdown while writing the first section in a journal file.
      2015-09-01T15:22:23.481+0200 [initandlisten] recover error: abrupt end to file /var/lib/mongodb/journal/j._12, yet it isn't the last journal file
      2015-09-01T15:22:23.482+0200 [initandlisten] dbexception during recovery: 13535 recover abrupt journal file end
      2015-09-01T15:22:23.487+0200 [initandlisten] exception in initAndListen: 13535 recover abrupt journal file end, terminating
      2015-09-01T15:22:23.487+0200 [initandlisten] dbexit:
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to close listening sockets...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to flush diaglog...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to close sockets...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: waiting for fs preallocator...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: lock for final commit...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: final commit...
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: closing all files...
      2015-09-01T15:22:23.487+0200 [initandlisten] closeAllFiles() finished
      2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: removing fs lock...
      2015-09-01T15:22:23.487+0200 [initandlisten] dbexit: really exiting now
      

      In this particular case, there was no data, so it was trivial to fix the problem by wiping the data directory and restarting the service. However, we obviously have some concerns about this.

      How likely is this to happen again? How often is mongod 'writing the first section of a journal file'?
      It this something that might be fixed in a later version?
      How do we restore normal operation, without losing data? Would it be a good idea to automate this with something like a 'watchdog'?

      Attachments

        Activity

          People

            kelsey.schubert@mongodb.com Kelsey Schubert
            robinp Robin Pedersen
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: