When I use the init script to restart the server daemon (i.e. /etc/init.d/mongodb restart), I often find that it starts recovering from the journal. Is the server not shutting down cleanly for some reason? See the log below:
Mon May 16 16:30:36 [dur] lsn set 514991446
Mon May 16 16:31:36 [dur] lsn set 515021610
Mon May 16 16:32:36 [dur] lsn set 515051824
Mon May 16 16:33:36 [dur] lsn set 515079249
Mon May 16 16:33:52 got kill or ctrl c or hup signal 15 (Terminated), will terminate after current cmd ends
Mon May 16 16:33:52 [interruptThread] now exiting
Mon May 16 16:33:52 dbexit:
Mon May 16 16:33:52 [interruptThread] shutdown: going to close listening sockets...
Mon May 16 16:33:52 [interruptThread] closing listening socket: 6
Mon May 16 16:33:52 [interruptThread] closing listening socket: 7
Mon May 16 16:33:52 [interruptThread] closing listening socket: 8
Mon May 16 16:33:52 [interruptThread] closing listening socket: 10
Mon May 16 16:33:52 [interruptThread] removing socket file: /tmp/mongodb-27018.sock
Mon May 16 16:33:52 [interruptThread] removing socket file: /tmp/mongodb-28018.sock
Mon May 16 16:33:52 [interruptThread] shutdown: going to flush diaglog...
Mon May 16 16:33:52 [interruptThread] shutdown: going to close sockets...
Mon May 16 16:33:52 [interruptThread] shutdown: waiting for fs preallocator...
Mon May 16 16:33:52 [interruptThread] shutdown: lock for final commit...
Mon May 16 16:33:52 [interruptThread] shutdown: final commit...
Mon May 16 16:33:52 ERROR: Client::shutdown not called: slaveTracking
Mon May 16 16:33:52 [interruptThread] shutdown: closing all files...
Mon May 16 16:33:52 [replica set sync] MessagingPort recv() errno:9 Bad file descriptor 10.84.55.204:27018
Mon May 16 16:33:52 [replica set sync] SocketException: remote: error: 9001 socket exception [1]
Mon May 16 16:33:52 [replica set sync] MessagingPort flush send() errno:9 Bad file descriptor 10.84.55.204:27018
Mon May 16 16:33:52 [replica set sync] caught exception (socket exception) in destructor (~PiggyBackData)
Mon May 16 16:33:52 [replica set sync] replSet syncThread: 10278 dbclient error communicating with server: mongodb04.example.com:27018
37/60 61%
46/60 76%
Mon May 16 16:33:58 [initandlisten] MongoDB starting : pid=27843 port=27018 dbpath=/var/lib/mongodb 64-bit
Mon May 16 16:33:58 [initandlisten] db version v1.8.2-rc2-pre-, pdfile version 4.5
Mon May 16 16:33:58 [initandlisten] git version: 78644abdfb4a7ca834e8bdeaed55cde51989de12
Mon May 16 16:33:58 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Mon May 16 16:33:58 [initandlisten] journal dir=/var/lib/mongodb/journal
Mon May 16 16:33:58 [initandlisten] recover begin
Mon May 16 16:33:58 [initandlisten] recover lsn: 0
Mon May 16 16:33:58 [initandlisten] recover /var/lib/mongodb/journal/j._965
Mon May 16 16:34:12 [initandlisten] recover /var/lib/mongodb/journal/j._966
Mon May 16 16:34:17 [initandlisten] recover cleaning up
Mon May 16 16:34:17 [initandlisten] removeJournalFiles
Mon May 16 16:34:17 [initandlisten] recover done