Details
-
Question
-
Resolution: Cannot Reproduce
-
Major - P3
-
None
-
2.6.5
-
None
Description
We are currently deploying MongoDB on a number of sites, running Ubuntu 12.04 precise. On one of these systems, after running MongoDB for a few weeks, we discovered that the 'mongod' service had stopped. Trying to start it gave the following log:
2015-09-01T15:22:23.447+0200 ***** SERVER RESTARTED *****
|
2015-09-01T15:22:23.463+0200 [initandlisten] MongoDB starting : pid=31061 port=27017 dbpath=/var/lib/mongodb 64-bit host=intelecom-edda-ferd-gondan-pcie-cb1
|
2015-09-01T15:22:23.463+0200 [initandlisten] db version v2.6.5
|
2015-09-01T15:22:23.463+0200 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
|
2015-09-01T15:22:23.463+0200 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
|
2015-09-01T15:22:23.463+0200 [initandlisten] allocator: tcmalloc
|
2015-09-01T15:22:23.463+0200 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
|
2015-09-01T15:22:23.464+0200 [initandlisten] journal dir=/var/lib/mongodb/journal
|
2015-09-01T15:22:23.464+0200 [initandlisten] recover begin
|
2015-09-01T15:22:23.471+0200 [initandlisten] recover lsn: 2811296823
|
2015-09-01T15:22:23.471+0200 [initandlisten] recover /var/lib/mongodb/journal/j._12
|
2015-09-01T15:22:23.479+0200 [initandlisten] Journal file header invalid. This could indicate corruption, or an unclean shutdown while writing the first section in a journal file.
|
2015-09-01T15:22:23.481+0200 [initandlisten] recover error: abrupt end to file /var/lib/mongodb/journal/j._12, yet it isn't the last journal file
|
2015-09-01T15:22:23.482+0200 [initandlisten] dbexception during recovery: 13535 recover abrupt journal file end
|
2015-09-01T15:22:23.487+0200 [initandlisten] exception in initAndListen: 13535 recover abrupt journal file end, terminating
|
2015-09-01T15:22:23.487+0200 [initandlisten] dbexit:
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to close listening sockets...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to flush diaglog...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: going to close sockets...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: waiting for fs preallocator...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: lock for final commit...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: final commit...
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: closing all files...
|
2015-09-01T15:22:23.487+0200 [initandlisten] closeAllFiles() finished
|
2015-09-01T15:22:23.487+0200 [initandlisten] shutdown: removing fs lock...
|
2015-09-01T15:22:23.487+0200 [initandlisten] dbexit: really exiting now
|
In this particular case, there was no data, so it was trivial to fix the problem by wiping the data directory and restarting the service. However, we obviously have some concerns about this.
How likely is this to happen again? How often is mongod 'writing the first section of a journal file'?
It this something that might be fixed in a later version?
How do we restore normal operation, without losing data? Would it be a good idea to automate this with something like a 'watchdog'?