[SERVER-4647] config server not started give error "there are no journal files to recover." Created: 08/Jan/12  Updated: 15/Aug/12  Resolved: 22/Apr/12

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.0.2
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: jitendra Assignee: Richard Kreuter (Inactive)
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux debian


Operating System: Linux
Participants:

 Description   

we run one mongos , one config server and 10 mongod shards.
we kill all mongo possess and restart all server. but config server not start and give error "there are no journal files to recover."

after this run config server with --repair. and start mongos , insert one record it give socket exception.

logs when config server not start.

old lock file: /u01/shard1/configdb/mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
found 1 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************
Sun Jan 8 19:00:01 [initandlisten] User Assertion: 12596:old lock file
Sun Jan 8 19:00:01 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
Sun Jan 8 19:00:01 dbexit:
Sun Jan 8 19:00:01 [initandlisten] shutdown: going to close listening sockets...
Sun Jan 8 19:00:01 [initandlisten] shutdown: going to flush diaglog...
Sun Jan 8 19:00:01 [initandlisten] shutdown: going to close sockets...
Sun Jan 8 19:00:01 [initandlisten] shutdown: waiting for fs preallocator...
Sun Jan 8 19:00:01 [initandlisten] shutdown: lock for final commit...
Sun Jan 8 19:00:01 [initandlisten] shutdown: final commit...
Sun Jan 8 19:00:01 [initandlisten] shutdown: closing all files...
Sun Jan 8 19:00:01 [initandlisten] closeAllFiles() finished
Sun Jan 8 19:00:01 dbexit: really exiting now

log when insert a record

Sun Jan 8 20:19:53 [conn2] creating new connection to:192.168.50.168:10000
Sun Jan 8 20:19:53 BackgroundJob starting: ConnectBG
Sun Jan 8 20:19:53 [conn2] DBException in process: socket exception
Sun Jan 8 20:19:58 [conn2] Socket recv() conn closed? 192.168.50.51:63406
Sun Jan 8 20:19:58 [conn2] SocketException: remote: 192.168.50.51:63406 error: 9001 socket exception [0] server [192.168.50.51:63406]
Sun Jan 8 20:20:20 [conn3] Request::process ns: 00080112.Database msg id:41 attempt: 0
Sun Jan 8 20:20:20 [conn3] write: 00080112.Database



 Comments   
Comment by Ian Whalen (Inactive) [ 11/Apr/12 ]

@jitendra, could you please attach the config log files?

Comment by Richard Kreuter (Inactive) [ 10/Jan/12 ]

Two things:

(1) could you upload your entire config database's log file, from before and after the initial kill operation?

(2) I can't tell whether you've got your config server running again. Have you?

Comment by jitendra [ 10/Jan/12 ]

yes i was running with journalling and didn't delete the journal/ directory?

Comment by Eliot Horowitz (Inactive) [ 10/Jan/12 ]

Are you sure you were running with journalling before and now and that you didn't delete the journal/ directory?

Comment by jitendra [ 10/Jan/12 ]

this error come then what to do?
**************
old lock file: mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
found 23 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************

Comment by Eliot Horowitz (Inactive) [ 10/Jan/12 ]

You should check that your startup scrips do not remove the lock files or any files.
This shouldn't happen unles you're removing some files.

Comment by jitendra [ 10/Jan/12 ]

this thing happened with one mongod shard server too, that time disk was full.

Comment by Eliot Horowitz (Inactive) [ 10/Jan/12 ]

You should really really be running 3 config servers.
In that case you can dump from one of those.

If you just have 1 - I would run a repair.

Comment by jitendra [ 10/Jan/12 ]

we started server same way.
if this problem come then what should do.

Comment by Eliot Horowitz (Inactive) [ 09/Jan/12 ]

That can happen if you don't start a server the same way.
Is that possible?

Comment by jitendra [ 09/Jan/12 ]

if journal enable then how to handle below problem. why does it come?

**************
old lock file: mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
found 23 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************

You may want to check:

If someone moved the journal files
The integrity of your disk.

Comment by Eliot Horowitz (Inactive) [ 09/Jan/12 ]

Did you start the config server the same way as you did the first time?

Comment by Scott Hernandez (Inactive) [ 08/Jan/12 ]

Please run db.serverCmdLineOpts(), and db.serverStatus() on the config server.

Generated at Thu Feb 08 03:06:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.