[SERVER-69485] Profiler setting of low "slowms" creates huge log file Created: 07/Sep/22  Updated: 27/Oct/23  Resolved: 15/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Joel J Assignee: Yuan Fang
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-18946 I wish I can configure separately the... Open
Operating System: ALL
Participants:

 Description   

When updating the profiler settings to log queries profiling info into the collection system.profile, the mongod.log file is recording every query in the text file.

 

This leads to filling the hard drive very fast, for nothing. I already have all the information in the collection, that i can query using filters.

 

I am forced to disable the profiling or to set the slowQuery threshold to a high value, otherwise the hard drive will be full very fast because of mongod.log. The settings of the mongod.log profiling and the settings of system.profile profiling should not be the same setting. It's not the same thing. The system.profile is a rotating log that can never fill the hard drive.

 

The system.profile collection can be used even with fast queries, to determine the query plans and search coll scans, in memory sorts, and bad indexes. Even if the query is super fast with a small data sample.

 

It seems impossible to use as it is. I have disabled it, though i need profiling, but without filling the hard drive with mongod.log detailed query logs.

 

Regards.



 Comments   
Comment by Yuan Fang [ 14/Sep/22 ]

Hi,

Thank you for your report. I understand that you observed the mongod.log being filled quickly when the database profiler is enabled. I believe the reason is that the "slowms" (i.e. slowQuery threshold) configuration setting affects the behavior of both the diagnostic log (i.e. mongod.log) and profiler (system.profile).

In order to reduce the number of slow queries written to mongod.log, you will need to increase the "slowms" to a reasonably high value. See MongoDB Doc - db.setProfilingLevel()for details. Also, consider triggering Rotate Log Files to keep the log file size below a predetermined threshold, which might be helpful with the disk space issues.

As importantly, I can see the feature of being able to config the profiler and diagnostic log independently would be valuable for making MongoDB easier to work with. However, this JIRA project is for bug reports, but you can help get this on the radar by submitting a feature request to our MongoDB Feedback Engine.

For further issues, we'd like to encourage you to start by asking our community for help by posting on the MongoDB Developer Community Forums. If the discussion there leads you to suspect a bug in the MongoDB server, then we'd want to investigate it as a possible bug here in the SERVER project.

Regards,

Yuan

Generated at Thu Feb 08 06:13:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.