Summary
Allow users to set a custom filter that determines which operations are written to the mongod/mongos log files and system.profile collection. Add support for operation sampling within the filter.
Motivation
Identifying operations that cause performance degradation is a core part of the MongoDB diagnostic experience. These operations can be reviewed manually by users in the logs or via tools such as Visual Query Profiler in Atlas/OM/CM. They are also utilized by Performance Advisor to automatically recommend what indexes should be created and in the future can be used by Schema Advisor to recommend schema improvements.
Today, we only log operations over 100ms by default, which is often insufficient to holistically understand the impact of operations on performance and to make actionable recommendations on. Via the profile command, a user can configure any mongod to have a different threshold for slow queries or to log a sample of all operations. The controls we have over what is logged is coarse and we are unable to capture all the slow operations as well in that scenario.
We want to provide our users with a way to granularly control what operations are logged, so that their diagnostic experience will be enhanced. In particular, Performance Advisor would like to specify a filter that logs both slow operations as well as a 1% sample of all operations. This is not possible today because the "sampleRate" parameter samples exclusively from operations which exceed the slowms threshold.
Additionally, custom filters will allow us greater flexibility when capturing workloads from production systems with the goal of creating benchmarks or making product decisions. They will allow us to capture specific workloads or namespaces, while also being capable of implementing the same behaviour that slowms and sampleRate offer currently.
Cast of Characters
- Product Owner: Katya Kamenieva
- Project Lead: David Percy
- Program Manager: Craig Homa
Documentation
Scope Document
Technical Design Document