[SERVER-40756] Always log storage watchdog kills to syslog Created: 20/Apr/19 Updated: 29/Oct/19 Resolved: 01/May/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Cailin Nelson | Assignee: | Andrew Morrow (Inactive) |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Sprint: | Dev Tools 2019-05-06 | ||||
| Participants: | |||||
| Description |
|
Consider a configuration in which the mongod log files are written to the same partition as the mongod data files. This configuration tends to result in the following behavior when the storage watchdog triggers: a) the partition is non-functional in some way b) the storage watchdog triggers and kills the mongod so c) the fact that the storage watchdog killed the mongod does not appear in the log files. Trying to figure out what killed the mongod in these situations is non-trivial. Is it possible to have the storage watchdog kills logged in some other way? For example, could just that one log message also be sent to syslog - even if syslog logging is not enabled? When a mongod process and it's log file end abruptly, I think most experienced users are accustomed to going to look at the syslog to find evidence of an OOM, or other hardware issue. This would be an intuitive place to capture the storage watchdog kills. |
| Comments |
| Comment by Kevin Pulo [ 01/May/19 ] |
|
The process does exit with a unique exit code, EXIT_WATCHDOG, defined as 61, which could be checked by the parent when the process dies. If the server was started without --fork, then this is straightforward. Otherwise if --fork was used, then it would need to rely on the pid 1 init process for this check (which may or may not be possible to configure, depending on the init system in use). acm's point is still valid — depending on how hosed the system is, it may not be possible for the parent (init or otherwise) to persistently record the fact that the watchdog triggered. However, it might be possible to better handle partial system failures this way, and would certainly be more flexible than anything we could bake into the server. |
| Comment by Andrew Morrow (Inactive) [ 26/Apr/19 ] |
|
Unfortunately, this behavior is an explicit and intentional design decision. Once the watchdog notices that the system is unhealthy, it needs to terminate immediately, because it can't know whether any other operations are safe to execute, in the sense that they won't hang. Adding any intervening actions between the detection of a bad state and termination risks making the watchdog ineffective. We can really only hope that the _exit() syscall will work, and we must assume that anything else may trigger disk activity. |