[SERVER-6896] Server crashes if logRotate is issued and the file cannot be rotated Created: 30/Aug/12 Updated: 15/Feb/13 Resolved: 06/Dec/12 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability |
| Affects Version/s: | 2.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Sridhar Nanjundeswaran | Assignee: | Tad Marshall |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
mongodb v2.2 2008+ build on Windows 7 x64 |
||
| Issue Links: |
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Start db with logpath. From a command prompt do more c:\data\mongod.log Now connect using the mongo shell and do db.adminCommand("logRotate"). This causes the server to die with the following in the logs
|
| Comments |
| Comment by Tad Mannes [ 10/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hello Dan, the permissions on /home/bgtxatl/data_store/logs is: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Daniel Pasette (Inactive) [ 08/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Can you send the permissions on /home/bgtxatl/data_store/logs/? | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Mannes [ 07/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I neglected to mention that we run on RedHat EL 6 in production, and we built our own mongo executables, libraries, from source. The source we used was actually version 2.2.1 so this issue is clearly not resolved. Tue Dec 4 16:54:17 [initandlisten] journal dir=/home/bgtxatl/data_store/database/journal | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Mannes [ 07/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
We also get this issue. What is interesting is that we can issue the logRotate command by hand and it generally works, but when we put it in a bash script and it runs near midnight GMT, we frequently see this issue. At present there is very little to no usage of the database at so I don't believe the issue is related to it being in use. Our bash script basically does this: echo "db.runCommand( {logRotate:1});" | $MONGODIR/bin/mongo $BADGER_DATASTORE_SERVER/admin 2>&1 Here is the very end of our mongod log file: ***aborting after fassert() failure Fri Dec 7 00:05:02 Got signal: 6 (Aborted). Fri Dec 7 00:05:02 Backtrace: We do log rotation every night, and this is becoming a serious issue for us. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ian Whalen (Inactive) [ 06/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
this should indeed fassert() as it does | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Marshall [ 04/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Fatal Assertions 16175 and 16176 are both for a failure of the 'rotateLogs()' function, but come from two different places in the code. 16175 is for a failure of the 'logRotate' command (e.g. 'db.adminCommand("logRotate")'). 16176 is for a failure when SIGUSR1 has been received (not available on Windows). The triggering failure (for this case) of the rename operation is the same for both Fatal Assertions. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Kristian Brimble [ 04/Dec/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Same failure conditions on 12:04 but looks like it was a different assertion.
This has only happened once. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tony Nelson [ 24/Sep/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I am having the same problem on Ubuntu 10.04 (lucid). This server is actually unused at the moment, but every Sunday when logrotate executes, the server crashes. Here is log for the last week:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Sridhar Nanjundeswaran [ 30/Aug/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Definitely not a fatal assert. In fact we seem to be handling it on startup. So have a large mongod.log file on which you do a more. Now start mongod with logpath pointing to this file. In this scenario we just log to console and start up. I see this at the beginning of the console log | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Tad Marshall [ 30/Aug/12 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Also, the error reporting doesn't report the OS error number and string, and is missing an endl so the Fatal Assertion text is on the same line. I don't think this should be a fatal assertion, since it does not indicate corrupted state. |