[SERVER-18339] Log only warning, error or fatal messages Created: 06/May/15  Updated: 08/Jan/24  Resolved: 18/Mar/16

Status: Closed
Project: Core Server
Component/s: Admin, Logging
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Andy Walker Assignee: Mira Carey
Resolution: Done Votes: 1
Labels: log-verbosity
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-44866 Impossible to custom log verbosity to... Closed
Related
related to SERVER-35035 Additional levels of command debugging Backlog
Backwards Compatibility: Fully Compatible
Participants:

 Description   

We have lately started using syslog to hold the mongodb logs. However, it is a little overwhelming within the amount of informational logs we receive.

Also, since enabling the --syslog flag the system has decreased in performance

The below links describe, 0 is the default verbosity level and includes informational logs. How 1 - 5 only increases the verbosity level, is there a way to only log warning, error or fatal messages?

http://docs.mongodb.org/manual/reference/method/db.setLogLevel/

http://docs.mongodb.org/manual/reference/log-messages/



 Comments   
Comment by Paul Hollas [ 15/Aug/18 ]

Hi, please could you consider looking at re-opening this. I have cases where sometimes we need to shut down secondary nodes for say 48-72 hours for a planned maintenance activity and the logs on the surviving nodes are spammed by network errors and there is no current way to suppress them. Even with log rotate and compression the logs can easily fill a file system and affect availability of surviving data nodes. Yes we could reconfigure a replica set and remove the nodes to stop this and indeed if an outage was going to be prolonged ie generally longer than the oplog window size, we would take this route and opt for a node initial sync.  However, it would be very useful to be able to radically supress logging dynamically without having to do a replica set reconfig and of course support teams would have to live with the consequences of no logging. Have to add I work with many other database technologies and I laud to stake holders all the time about how good MongoDBs reliability is.  

Many thanks

Paul

Comment by Judah Schvimer [ 14/Oct/16 ]

Hi lpolonen,

On further review and testing, it looks like providing --quiet will not suppress replication log messages unless you are using master/slave replication: https://docs.mongodb.com/v3.2/core/master-slave/.

Comment by Lassi Pölönen [ 14/Oct/16 ]

Any chance this could be reopened? With for example PHP applications it's next to impossible to have persistent long lived connections as it's very common to limit the amount of requests served by the workers for good reasons so with a lot of clients there are always going to be a lot of new connections. So there are cases where re-use is not possible to the same extent. Quiet mode just can't go to production due to lack of replication logging.

Comment by Lassi Pölönen [ 02/Jun/16 ]

Thought about it. Thought the replication logging activities are important on especially resync when there are the highest amount of connection attempts. Those are the moments when the amount of logging is the biggest issue.

Comment by Mira Carey [ 02/Jun/16 ]

lpolonen,

A colleague reminded me about --quiet. While it's generally not recommended for production systems (because connection information is very helpful for debugging), it does suppress connection logging.

The full list of events --quiet suppresses:

  • connection events;
  • the drop command, the dropIndexes command, the diagLogging command, the validate command, and the clean command; and
  • replication synchronization activities.
Comment by Mira Carey [ 02/Jun/16 ]

lpolonen,

To clarify, you're generating 1k of logs per second entirely or mostly due to clients connecting and disconnecting? We haven't considered connections and disconnections to be a frequent enough event to warrant hiding behind a higher verbosity setting because connections are generally expected to be long lived or infrequent. If these log lines are occurring very often for you, it would indicate that you either have a very large number of clients (so large that even though connections are long lived, a substantial number are connecting and disconnecting at any one time) or that you're getting relatively little connection re-use (your client code may be connecting, performing a small number of operations and immediately disconnecting). If the latter case is true, you may want to look into connection re-use in your application, as you may be experiencing more latency per operation than you really need to.

Regards,
Jason

Comment by Lassi Pölönen [ 01/Jun/16 ]

Would be much appreciated if the logging of connections could be reduced. Getting about 1k per second of:

2016-06-01T23:09:38.703+0000 I NETWORK [initandlisten] connection accepted from ...
2016-06-01T23:09:38.775+0000 I NETWORK [conn1199019493] end connection ..

Sure there are many clients connecting, but those messages mostly fill disk and don't provide much useful information.

Comment by hydrapolic [ 08/Mar/16 ]

Yes Jason, after setting slowOpThresholdMs, the logs are ok.

Comment by Mira Carey [ 07/Mar/16 ]

hydrapolic,

Are your logs still too large after tuning up the slowOpThresholdMs? If not, I'd like to close this out as works as designed.

Regards,
Jason

Comment by hydrapolic [ 28/Jul/15 ]

Yes you are right, they all took more than 100ms. After adjusting that value the messages are gone.

Comment by Daniel Pasette (Inactive) [ 28/Jul/15 ]

I can't be sure unless you include full log lines, but I'm assuming those operations are exceeding the operation time limit threshold. It is 100ms by default. It can be adjusted by setting this configuration: http://docs.mongodb.org/manual/reference/configuration-options/#operationProfiling.slowOpThresholdMs

Comment by hydrapolic [ 28/Jul/15 ]

The logging is set like:

systemLog:
destination: file
path: "/var/log/mongodb/mongodb.log"
quiet: true
logAppend: true

Comment by hydrapolic [ 28/Jul/15 ]

We use mapReduce a lot so the lines look like:
2015-07-28T03:10:27.204+0000 I COMMAND [conn1727] command db.$cmd command: mapReduce ...

Or information about writes
2015-07-28T03:11:17.350+0000 I WRITE [conn18080] insert ...
2015-07-28T03:13:08.019+0000 I WRITE [conn57] update ...

Comment by Daniel Pasette (Inactive) [ 27/Jul/15 ]

hydrapolic, can you tell me if there is a particular (or set of) informational log message that is filling up your log files?

The problem with disabling informational messages is that debugging problems with replication and sharding become very difficult without those log entries. Under normal circumstances, the volume of messages should not be overwhelming, and thus it may be a symptom of an underlying problem.

Comment by hydrapolic [ 27/Jul/15 ]

Please reconsider this task. We are running a MongoDB 2.4 cluster that only has a few MB of logs during the day and a MongoDB 3.0 cluster with GBs of logs each day that are pretty useless. Please allow us to disable logging of informational log messages so we are not lost when looking for warnings/errors.

Comment by Ramon Fernandez Marina [ 21/May/15 ]

andy@ditno.com, I forgot to mention that you may want to consider the user of logrotate to periodically rotate and compress logs – hope this helps with the size of the logs you need to keep.

Regards,
Ramón.

Comment by Andy Walker [ 12/May/15 ]

Hi Ramon

we will be interested if the logging is reduced as we are curenttly not using syslog and dev/null

I have raised another ticket regarding performance, as this is beginning to be a huge problem - SERVER-18425

Thanks
Andy

Comment by Ramon Fernandez Marina [ 07/May/15 ]

andy@ditno.com, currently the lowest verbosity level is 0, and you're correct that the amount of logs only goes up from that. Two things about your report:

  1. I'm re-purposing your ticket as an improvement suggestion to have a less verbose log. While it sounds like an attractive idea, we have to take into account the impact that lowering the amount of logs may have on diagnostics and troubleshooting.
  2. If you're using a modern syslog you may want to investigate using filters for the logs generated by MongoDB. Assuming you're using 3.0, you can use the different severity levels to decide what to filter in/out.

Finally, can you please elaborate on the decrease in performance you mention? If there's a significant performance impact from using syslog we'd like to understand what's happening.

Thanks,
Ramón.

Generated at Thu Feb 08 03:47:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.