[SERVER-17862] Unify stats reporting for log/profiler across CRUD operations Created: 02/Apr/15  Updated: 18/Jul/17  Resolved: 09/Jun/16

Status: Closed
Project: Core Server
Component/s: Diagnostics, Logging
Affects Version/s: None
Fix Version/s: 3.3.9

Type: Improvement Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: James Wahlin
Resolution: Done Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documented
is documented by DOCS-10545 Docs for SERVER-17862: Unify stats re... Closed
Duplicate
is duplicated by SERVER-25657 Inconsistent getMore logging Closed
is duplicated by SERVER-3444 Alter profiler to capture nscanned wh... Closed
Related
related to SERVER-21855 Certain commands are never captured i... Backlog
related to SERVER-8912 Metric "document.scanned" is not the ... Closed
related to SERVER-15239 serverStatus metrics incorrect: "oper... Closed
related to SERVER-15240 serverStatus metric "document.returne... Closed
related to SERVER-16071 The nscanned and nscannedObjects is n... Closed
related to SERVER-23009 CRUD commands / operations should onl... Closed
related to SERVER-23255 Log keyUpdates & writeConflicts only ... Closed
related to SERVER-23081 keyUpdates not set on MMAPv1 document... Closed
related to SERVER-15102 dbCurrentOp should show used index Closed
related to SERVER-13034 Rename "upserted" field to "inserted"... Backlog
related to SERVER-23620 Add a failpoint which forces query re... Backlog
related to SERVER-23861 Add keysInserted & keysDeleted to ser... Backlog
related to SERVER-3129 Include projection in the slow query ... Closed
is related to SERVER-22250 Diagnostic log messages missing debug... Backlog
is related to SERVER-22252 Diagnostic log messages missing debug... Backlog
is related to SERVER-23357 db.serverStatus.metrics.document.inse... Closed
is related to SERVER-13419 Plan summary string in slow query log... Closed
is related to SERVER-3638 Full insert logging in query logs and... Closed
is related to SERVER-14432 Flexible logging of certain queries Closed
is related to SERVER-5830 Show original queries in currentOp or... Closed
is related to SERVER-5271 Add keysExamined, docsExamined, and n... Backlog
is related to SERVER-5834 Log final count for count command Backlog
is related to SERVER-7903 When logging a slow count, include th... Backlog
is related to SERVER-16265 Add query details to getmore entry in... Closed
is related to SERVER-16847 Diagnostic log output for all query c... Closed
is related to SERVER-17275 Remove "stats" field from distinct co... Closed
is related to SERVER-12823 Include query related fields for dele... Closed
is related to SERVER-21470 Add index usage information to getmor... Closed
Backwards Compatibility: Major Change
Sprint: Query 15 (06/03/16), Query 16 (06/24/16)
Participants:

 Description   
Issue Status as of Jun 08, 2016

ISSUE SUMMARY
This project improves the existing MongoDB diagnostics facilities by ensuring that the same diagnostic information is available across all operations.

Consistent reporting of metrics across user operations

All relevant metrics are supported for each command and operation (with a small list of exceptions below). These metrics are available in the slow diagnostic log, the profiler and db.currentOp().

Operations included

  1. count
  2. distinct
  3. find (command + OP_QUERY)
  4. findAndModify
  5. getMore (command + OP_GET_MORE)
  6. geoNear
  7. group
  8. mapReduce
  9. delete (command + OP_DELETE)
  10. update (command + OP_UPDATE)

Metrics covered

  1. keysExamined
  2. docsExamined
  3. reslen
  4. nreturned
  5. ntoreturn
  6. batchSize
  7. numYields
  8. cursorExhausted
  9. planSummary
  10. cursorid
  11. writeConflicts
  12. hasSortStage
  13. fromMultiPlanner
  14. replanned
  15. locks
  16. protocol
  17. millis
  18. ninserted (or 'upsert' for updates that result in an insert)
  19. multi
  20. nMatched
  21. nModified
  22. ndeleted
  23. ntoskip
  24. exhaust
  25. nmoved
  26. upsert
  27. execStats

Exceptions

  1. execStats is only available in the profiler output. It is not available in the slow diagnostics log or db.currentOp().
  2. execStats is not available for the aggregate command.
Introduction of new metrics

This project introduces 2 new metrics, 'keysInserted' and 'keysDeleted'. These metrics report the number of index keys inserted and deleted for all write operations. This replaces 'keyUpdates' which was not previously covered across write commands and is potentially misleading.

Removed metrics

This project removes the following metrics:

  1. fastmod: replaced by 'keysInserted', 'keysDeleted' and 'nmoved', which in combination can be used to derive this metric in a more granular, easy to understand manner.
  2. keyUpdates: replaced by 'keysInserted' and 'keysDeleted'.
  3. idhack: replaced by 'planSummary', which will report when the IDHACK stage is used.
  4. moved: obsoleted by the 'nmoved' metric.

See SERVER-23272 for the rationale behind this removal.

Deprecated metrics
  1. fastmodinsert: not considered generally useful for performance analysis.
Original description
Goals
  • Improve existing diagnostics by ensuring that the same information is available across operations (e.g. we don't report planSummary for all operations).
  • In other words, expose existing debug information more consistently.
Non-goals
  • Collect additional debug information which we don't already collect.
  • Major infrastructural changes to CurOp, the profiler, or the slow query log.
How

Part of the work for SERVER-10448 (a.k.a. "new explain"), completed for version 3.0, was to add a suite of functions capable of pulling summary statistics and debug information out of PlanExecutor (see explain.h). However, we use these functions in only a few places instead of uniformly populating CurOp with the available debug information. Using explain functionality to add the same stats to CurOp everywhere should be an easy win for diagnosability.

Details

This includes work for the following operations:

  • count
  • delete
  • distinct
  • find (OP_QUERY and command)
  • findAndModify
  • geoNear
  • getMore (OP_GET_MORE and command)
  • group
  • update

The following fields will be added to CurOp for every operation once the operation completes:

  • docsExamined
  • hasBlockingSort
  • idhack
  • keysExamined
  • nReturned
  • planSummary


 Comments   
Comment by Githook User [ 18/Mar/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-17862 Log keyUpdates & writeConflicts only when meaningful
Branch: master
https://github.com/mongodb/mongo/commit/7e8d5536f8a94fc5fd74b3b2027a9d045929117c

Comment by J Rassi [ 15/Jun/15 ]

And, SERVER-8912 as well.

Comment by J Rassi [ 05/May/15 ]

I suggest we pull SERVER-3129 into this ticket.  Linking as related.

Generated at Thu Feb 08 03:45:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.