[DOCS-15970] [Server] Customer logs can be lost/suppressed when logging to SYSLOG Created: 21/Mar/23 Updated: 30/Oct/23 Resolved: 13/Jul/23 |
|
| Status: | Closed |
| Project: | Documentation |
| Component/s: | Server |
| Affects Version/s: | None |
| Fix Version/s: | Server_Docs_20231030 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Jeffrey Natiuk | Assignee: | Dachary Carey |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | quick-win, server-docs-bug-bash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Linux, logging to syslog |
||
| Participants: | |
| Days since reply: | 29 weeks, 6 days ago |
| Description |
|
When logging to syslog, messages are subject to the rules as defined in /etc/systemd/journald.conf. By default, this limits log message bursts to 1000 messages within a 30 second period. Any messages that exceed this limit are enumerated and suppressed by systemd-journal (Suppressed 6512 messages from /system.slice/mongodb-mms-automation-agent.service). When it rains, it pours, and any sufficiently large problem within a cluster can lose messages precisely when those messages are most necessary for troubleshooting purposes. Proposing that we include this limitation in our documentation when customers are logging to syslog, as well as a recommendation to increase the RateLimitBurst parameter in /etc/systemd/journald.conf to handle more messages. In a recent customer case, we saw 10k-15k messages being suppressed. Potential updates:
|
| Comments |
| Comment by Dachary Carey [ 13/Jul/23 ] |
|
I accidentally marked this as "DOCSP" instead of "DOCS" in the GitHub PR, so the automatic linking didn't work. This was resolved in PR3775, which is now merged: https://github.com/10gen/docs-mongodb-internal/pull/3775 |