[SERVER-14725] In 2.6, mongod does not resume writing to the log file when the disk for the log volume is full, and then the space is free up Created: 30/Jul/14  Updated: 21/Apr/20  Resolved: 21/Apr/20

Status: Closed
Project: Core Server
Component/s: Logging
Affects Version/s: 2.6.3
Fix Version/s: None

Type: Bug Priority: Minor - P4
Reporter: Linda Qin Assignee: Sara Golemon
Resolution: Done Votes: 1
Labels: move-sec, polish
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
is duplicated by SERVER-15065 Mongdb servers stopped logging Closed
Related
is related to SERVER-3759 filesystem ops may cause termination ... Closed
Operating System: ALL
Sprint: Security 2020-05-04
Participants:

 Description   

The log files are saved on a different volume than the data and journals.

In 2.6, when the log volume is full, it stop writing to the log file. However, when we free the disk space, it does not write to the log file any more. We have to run logrotate command will trigger it to continue writing to the log file. This is different than 2.2/2.4.

In 2.2 and 2.4, when the log volume is full, it also stops writing to the log file. When we free the disk space, it would continue writing to the log file like this:

Tue Jul 29 20:34:57.822 [conn1] insert test.docs ninserted:1 keyUpdates:0 locks(micrTue Jul 29 20:35:26.093 [journal] _groupCommit 
Tue Jul 29 20:35:26.093 [journal] _groupCommit upgrade



 Comments   
Comment by Eric Milkie [ 21/Apr/20 ]

To clarify, the shut down that Sara mentions will be an unclean shutdown (no final checkpoint taken), and the act of shutting down itself will be "in the dark" since it will be impossible to log a message about it. But no other unlogged actions should occur besides the shutdown itself.

Comment by Sara Golemon [ 21/Apr/20 ]

The current logging subsystem will now shut down a node with a full log filesystem rather than carry on processing unlogged requests.  This makes the subject of resumability rather moot.

This isn't quite what this ticket is asking for, but is arguably the better action to take by default so that no actions are taken "in the dark".

Comment by Abutalib Zaidi [ 28/Aug/14 ]

that worked for me! Thanks Ramon!

Comment by Ramon Fernandez Marina [ 28/Aug/14 ]

I'm able to work around the issue by running logRotate twice. The first time I run logRotate the mongod.log file is rotated and a new one is created, but if the filesystem is still full mongod still won't log anything. After I move the rotated log file to some other filesystem to make some space, doing a second logRotate allows mongod to resume logging.

CC amzaidi.

Generated at Thu Feb 08 03:35:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.