[SERVER-11087] mongod/mongos fatally asserts when rotating logs and log isn't in original location Created: 08/Oct/13 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Admin, Logging, Security, Stability |
| Affects Version/s: | 2.4.6 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Noah Hoffman | Assignee: | Backlog - Security Team |
| Resolution: | Unresolved | Votes: | 6 |
| Labels: | platforms-re-triaged | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Running mongo 2.4.6 across the deployment Shard Nodes (mongod): Config Nodes: App Nodes (mongos): |
||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Assigned Teams: |
Server Security
|
||||||||||||||||||||||||||||||||
| Sprint: | Security 2020-04-20, Security 2020-05-04, Security 2020-06-01, Security 2020-06-15, Security 2020-07-27, Security 2021-01-11, Security 2021-02-22, Security 2021-03-08, Security 2021-03-22, Security 2021-04-05 | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Description |
|
While implementing a log rotate script we found that if the script would move the mongo*.log file to a new location BEFORE sending the SIGUSER signal to the daemon triggering the daemon's own rotate functionality, a fatal assertion is thrown Stack Trace: |
| Comments |
| Comment by Bill Ryder [ 17/Dec/14 ] | |||||||||||||||||||||||
|
Good suggestions thanks Akshay. I'll use the create and not cp /dev/null to the log file. Much cleaner. Also I've put an answer here if you want to check it out since I could have made errors and used your name. I can remove it if you like. https://serverfault.com/questions/540423/mongodb-proper-way-to-rotate-logs/653095 | |||||||||||||||||||||||
| Comment by Akshay Kumar [ 17/Dec/14 ] | |||||||||||||||||||||||
|
That would work. You are essentially doing the same thing in postrotate with the cp. create is IMO cleaner and more robust. The nocompress is not required as compression happens after postrotate, I updated the comment. I just like to have the last rotated file uncompressed on most of my systems for various reasons. | |||||||||||||||||||||||
| Comment by Bill Ryder [ 17/Dec/14 ] | |||||||||||||||||||||||
|
Unfortunately the packages I'm using -
do not seem to support the logRotate option.
Assuming I've set it up correctly (which I might not have of course):
This logrotate and doesn't crash the server without that optoin - but there is a small window where a few log entries may be lost.
If I really wanted to keep the logs perfectly I'd switch to syslog logging. | |||||||||||||||||||||||
| Comment by Akshay Kumar [ 17/Dec/14 ] | |||||||||||||||||||||||
|
Bill as Andy suggested you might want to look at If you have an older version take a look at the logrotate script on The key here is to use the create directive to logrotate, without that you'll trigger a crash. So in summary, logrotate file below will do the following:
| |||||||||||||||||||||||
| Comment by Bill Ryder [ 17/Dec/14 ] | |||||||||||||||||||||||
|
Amazingly many answers on the net describing how to use logrotate will break mongod with this bug. They all suggest
or variations on this theme - they all seem to send the signal after logrotate has moved the log file out of the way so mongod will crash. | |||||||||||||||||||||||
| Comment by Laurent Glayal [ 02/Dec/14 ] | |||||||||||||||||||||||
|
Also affect 2.6.x ( | |||||||||||||||||||||||
| Comment by Heikki Rauhala [ 31/Oct/13 ] | |||||||||||||||||||||||
|
Part of the problem is that the 10gen packaged version for Ubuntu does not contain any logrotate scripts, and the most popular unofficial documentation, like http://stackoverflow.com/a/8396266/61175 propose logrotate scripts that end up crashing the server. | |||||||||||||||||||||||
| Comment by Heikki Rauhala [ 31/Oct/13 ] | |||||||||||||||||||||||
|
Affects mongodb version 2.4.7 (0161738abf06c1f067b56a465b706efd6f4bf2aa) | |||||||||||||||||||||||
| Comment by Noah Hoffman [ 21/Oct/13 ] | |||||||||||||||||||||||
|
@Andy - sorry for the delay in responding. Yes, you've summarized it correctly, it should abort in this case, but not throw a fatal error. The other issue you pointed me to, 4905, is also a nice to have and I will vote for it but it wasn't the primary motivation for logging this issue. Thanks! | |||||||||||||||||||||||
| Comment by Andy Schwerin [ 09/Oct/13 ] | |||||||||||||||||||||||
|
The server intentionally aborts when log rotation doesn't go as planned, so as not to have periods of unlogged operation. The scenario you describe in this bug probably shouldn't lead to a fatal error, however. Out of curiosity, are you looking for the behavior requested in |