[DOCS-15970] [Server] Customer logs can be lost/suppressed when logging to SYSLOG Created: 21/Mar/23  Updated: 30/Oct/23  Resolved: 13/Jul/23

Status: Closed
Project: Documentation
Component/s: Server
Affects Version/s: None
Fix Version/s: Server_Docs_20231030

Type: Task Priority: Major - P3
Reporter: Jeffrey Natiuk Assignee: Dachary Carey
Resolution: Fixed Votes: 0
Labels: quick-win, server-docs-bug-bash
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, logging to syslog


Participants:
Days since reply: 29 weeks, 6 days ago

 Description   

When logging to syslog, messages are subject to the rules as defined in /etc/systemd/journald.conf.  By default, this limits log message bursts to 1000 messages within a 30 second period.  Any messages that exceed this limit are enumerated and suppressed by systemd-journal (Suppressed 6512 messages from /system.slice/mongodb-mms-automation-agent.service).

When it rains, it pours, and any sufficiently large problem within a cluster can lose messages precisely when those messages are most necessary for troubleshooting purposes.

Proposing that we include this limitation in our documentation when customers are logging to syslog, as well as a recommendation to increase the RateLimitBurst parameter in /etc/systemd/journald.conf to handle more messages.  In a recent customer case, we saw 10k-15k messages being suppressed. 

Potential updates:

 



 Comments   
Comment by Dachary Carey [ 13/Jul/23 ]

I accidentally marked this as "DOCSP" instead of "DOCS" in the GitHub PR, so the automatic linking didn't work. This was resolved in PR3775, which is now merged: https://github.com/10gen/docs-mongodb-internal/pull/3775

Generated at Thu Feb 08 08:14:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.