[SERVER-37743] Include loglevel in log messages Created: 24/Oct/18 Updated: 29/Oct/23 Resolved: 15/Mar/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.10 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kelsey Schubert | Assignee: | Andrew Morrow (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||
| Sprint: | Dev Tools 2019-03-25 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Currently, there is no quick way to filter log files collected at a higher verbosity to only include lower verbosity messages. By (optionally) including the loglevel in logged message, higher verbosity logs could be quickly filtered to speed initial investigation into issues. I'm imagining that a call to LOG(3), would include a 3 in the logged message. |
| Comments |
| Comment by Bruce Lucas (Inactive) [ 25/Mar/19 ] | |||
|
To summarize the outcome: D messages now included the log level, and other categories include an extra space to make the columns align:
| |||
| Comment by Githook User [ 15/Mar/19 ] | |||
|
Author: {'name': 'Andrew Morrow', 'username': 'acmorrow', 'email': 'acm@mongodb.com'}Message: | |||
| Comment by Andrew Morrow (Inactive) [ 15/Mar/19 ] | |||
|
Marking this as a minor change because it could break log parsers that assume existing fixed-width fields for the first three fields of the log. Hopefully, most log parsers are more robust and can accept the additional whitespace character on the non-debug log levels. | |||
| Comment by Bruce Lucas (Inactive) [ 26/Feb/19 ] | |||
|
Of course! | |||
| Comment by Kelsey Schubert [ 25/Feb/19 ] | |||
|
I'd prefer keeping the digit verbosity information since that'll allow more specific filtering later. I think Bruce's concerns can be addressed using awk. For example,
| |||
| Comment by Henrik Edin [ 25/Feb/19 ] | |||
|
Ok. An alternative design that perhaps meets the stated goal better could be a server parameter that sets a threshold. All debug logs above the threshold would keep log with 'D' and everything at and below could use 'V' (for verbose). Then it would be simple to filter out everything above/below a certain verbosity. That has the added benefit that by default there would be no change to the debug log format. | |||
| Comment by Bruce Lucas (Inactive) [ 25/Feb/19 ] | |||
|
The use case mentioned in the opening comment is to filter out all message above a certain level, for all components, and that use case makes sense to me. If the digit is guaranteed to be in the same column on all lines there may be some way to use standard Linux tools to do filtering on the character in that position, but it escapes me at the moment. | |||
| Comment by Henrik Edin [ 25/Feb/19 ] | |||
|
If you add the component it would be greppable: "3 NETWORK" for example. But if you just want everything at debug level 3 it would be problematic, " 3 " is probably not super robust. Is that good enough? It would be nice to keep using just 1 character so the rest of the formatting doesn't change. But if we can do that we could use "D3" here or something. | |||
| Comment by Bruce Lucas (Inactive) [ 25/Feb/19 ] | |||
|
That doesn't seem readily greppable though, but maybe there's some other simple way to filter on log level using standard commands? | |||
| Comment by Kelsey Schubert [ 25/Feb/19 ] | |||
|
Replacing the D with the digit sounds like great solution to me! I think think this behavior should be hidden behind a flag especially if we're backporting it as it may break diagnostic tools that ingest logs (though I'm struggling to imagine how at the moment, maybe someone who spent more time building log analysis tools could chime in). In any case, I think it should be enabled by default going forward. | |||
| Comment by Henrik Edin [ 12/Feb/19 ] | |||
|
What do you mean by optionally? Do you want this functionality to be hidden behind a server parameter? Do you have any preference on the format? I was thinking to just replace the 'D' currently output for severity > 0 to be the digit instead. '1', '2', …, '5'. That would have minimal impact on the actual log format as the severity is still being represented by a single character. | |||
| Comment by Bruce Lucas (Inactive) [ 26/Oct/18 ] | |||
|
Would be ideal if we can come up with a way to do this that 1) doesn't make the log line too much longer and 2) can be filtered using a simple grep. |