[SERVER-15643] mongod aborts on log file rotation when log file is missing Created: 14/Oct/14  Updated: 30/Nov/14  Resolved: 30/Nov/14

Status: Closed
Project: Core Server
Component/s: Logging
Affects Version/s: 2.6.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Laurent Glayal Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-11087 mongod/mongos fatally asserts when ro... Backlog
Operating System: ALL
Steps To Reproduce:
  • rename mongodb.log file
  • send USR12 to server to force a log rotation
Participants:

 Description   

Hi,
using external logrotate configuration on 2.6.4 results in server crash .
the logrotate renames the initial mongodb.log file in another name then sends the USR1 signal to mongod, this one doesn't find initial file and so terminates.

2014-10-14T11:36:43.188+0200 [clientcursormon] connections:2
2014-10-14T11:38:45.790+0200 [signalProcessingThread] warning: Rotating log file /var/log/mongodb/mongodb.log failed: FileRenameFailed Failed to rename
"/var/log/mongodb/mongodb.log" to "/var/log/mongodb/mongodb.log.2014-10-14T09-38-45": No such file or directory (2)
2014-10-14T11:38:45.790+0200 [signalProcessingThread] Fatal Assertion 16782
2014-10-14T11:38:45.797+0200 [signalProcessingThread] 0x11e6111 0x1187e49 0x116a97d 0x11e57cf 0x122aaa9 0x7ffa481cb9d1 0x7ffa47570b5d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1187e49]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x116a97d]
/usr/bin/mongod() [0x11e57cf]
/usr/bin/mongod() [0x122aaa9]
/lib64/libpthread.so.0(+0x79d1) [0x7ffa481cb9d1]
/lib64/libc.so.6(clone+0x6d) [0x7ffa47570b5d]
2014-10-14T11:38:45.797+0200 [signalProcessingThread]

***aborting after fassert() failure

2014-10-14T11:38:45.802+0200 [signalProcessingThread] SEVERE: Got signal: 6 (Aborted).
Backtrace:0x11e6111 0x11e54ee 0x7ffa474ba9a0 0x7ffa474ba925 0x7ffa474bc105 0x116a9ea 0x11e57cf 0x122aaa9 0x7ffa481cb9d1 0x7ffa47570b5d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod() [0x11e54ee]
/lib64/libc.so.6(+0x329a0) [0x7ffa474ba9a0]
/lib64/libc.so.6(gsignal+0x35) [0x7ffa474ba925]
/lib64/libc.so.6(abort+0x175) [0x7ffa474bc105]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x13a) [0x116a9ea]
/usr/bin/mongod() [0x11e57cf]
/usr/bin/mongod() [0x122aaa9]
/lib64/libpthread.so.0(+0x79d1) [0x7ffa481cb9d1]
/lib64/libc.so.6(clone+0x6d) [0x7ffa47570b5d]

  • having a missing log file is frequent when people (or logrotate) delete log files expecting to free disk space

IMHO : even if original ile is missing the server should not terminate.

Previously I used mongo 2.4.6 from centos repositories with logrotate and had no trouble, I switched to mongo 2.6.4 from mongodb repositories and reused previous logrotate configuration, ad now my servers are crashing at log rotation.

Regards.



 Comments   
Comment by Ramon Fernandez Marina [ 30/Nov/14 ]

lglayal, this issue was reported before in SERVER-11087, so I'm marking this ticket as a duplicate. Please feel free to watch and vote for SERVER-11087. You may also be interested in SERVER-4905.

Regards,
Ramón.

Comment by Laurent Glayal [ 14/Oct/14 ]

Would it be possible to replace

case SIGUSR1:
// log rotate signal
fassert(16782, rotateLogs());
logProcessDetailsForLogRotate();
break;

with something like

case SIGUSR1:
// log rotate signal
if(! rotateLogs())

{ logProcessDetailsForLogRotate(); }

else

{ checkAnMayBeReopenLogFile(); logWarningOnLogrotate("Unable to rotate initial log file"); }

break;

Generated at Thu Feb 08 03:38:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.