[SERVER-40756] Always log storage watchdog kills to syslog Created: 20/Apr/19  Updated: 29/Oct/19  Resolved: 01/May/19

Status: Closed
Project: Core Server
Component/s: Logging
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Cailin Nelson Assignee: Andrew Morrow (Inactive)
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Sprint: Dev Tools 2019-05-06
Participants:

 Description   

Consider a configuration in which the mongod log files are written to the same partition as the mongod data files. This configuration tends to result in the following behavior when the storage watchdog triggers: a) the partition is non-functional in some way b) the storage watchdog triggers and kills the mongod so c) the fact that the storage watchdog killed the mongod does not appear in the log files. Trying to figure out what killed the mongod in these situations is non-trivial.

Is it possible to have the storage watchdog kills logged in some other way? For example, could just that one log message also be sent to syslog - even if syslog logging is not enabled? When a mongod process and it's log file end abruptly, I think most experienced users are accustomed to going to look at the syslog to find evidence of an OOM, or other hardware issue. This would be an intuitive place to capture the storage watchdog kills.



 Comments   
Comment by Kevin Pulo [ 01/May/19 ]

The process does exit with a unique exit code, EXIT_WATCHDOG, defined as 61, which could be checked by the parent when the process dies. If the server was started without --fork, then this is straightforward. Otherwise if --fork was used, then it would need to rely on the pid 1 init process for this check (which may or may not be possible to configure, depending on the init system in use).

acm's point is still valid — depending on how hosed the system is, it may not be possible for the parent (init or otherwise) to persistently record the fact that the watchdog triggered. However, it might be possible to better handle partial system failures this way, and would certainly be more flexible than anything we could bake into the server.

Comment by Andrew Morrow (Inactive) [ 26/Apr/19 ]

Unfortunately, this behavior is an explicit and intentional design decision. Once the watchdog notices that the system is unhealthy, it needs to terminate immediately, because it can't know whether any other operations are safe to execute, in the sense that they won't hang. Adding any intervening actions between the detection of a bad state and termination risks making the watchdog ineffective. We can really only hope that the _exit() syscall will work, and we must assume that anything else may trigger disk activity.

Generated at Thu Feb 08 04:55:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.