Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-15970

[Server] Customer logs can be lost/suppressed when logging to SYSLOG

    XMLWordPrintableJSON

Details

    Description

      When logging to syslog, messages are subject to the rules as defined in /etc/systemd/journald.conf.  By default, this limits log message bursts to 1000 messages within a 30 second period.  Any messages that exceed this limit are enumerated and suppressed by systemd-journal (Suppressed 6512 messages from /system.slice/mongodb-mms-automation-agent.service).

      When it rains, it pours, and any sufficiently large problem within a cluster can lose messages precisely when those messages are most necessary for troubleshooting purposes.

      Proposing that we include this limitation in our documentation when customers are logging to syslog, as well as a recommendation to increase the RateLimitBurst parameter in /etc/systemd/journald.conf to handle more messages.  In a recent customer case, we saw 10k-15k messages being suppressed. 

      Potential updates:

       

      Attachments

        Activity

          People

            dachary.carey@mongodb.com Dachary Carey
            jeff.natiuk@mongodb.com Jeffrey Natiuk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:
              29 weeks, 6 days ago