Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-15970

[Server] Customer logs can be lost/suppressed when logging to SYSLOG

      When logging to syslog, messages are subject to the rules as defined in /etc/systemd/journald.conf.  By default, this limits log message bursts to 1000 messages within a 30 second period.  Any messages that exceed this limit are enumerated and suppressed by systemd-journal (Suppressed 6512 messages from /system.slice/mongodb-mms-automation-agent.service).

      When it rains, it pours, and any sufficiently large problem within a cluster can lose messages precisely when those messages are most necessary for troubleshooting purposes.

      Proposing that we include this limitation in our documentation when customers are logging to syslog, as well as a recommendation to increase the RateLimitBurst parameter in /etc/systemd/journald.conf to handle more messages.  In a recent customer case, we saw 10k-15k messages being suppressed. 

      Potential updates:

       

            Assignee:
            dachary.carey@mongodb.com Dachary Carey
            Reporter:
            jeff.natiuk@mongodb.com Jeffrey Natiuk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:
              42 weeks, 1 day ago