[SERVER-23271] Add metrics to track index key additions and deletions for all CRUD ops/commands Created: 21/Mar/16  Updated: 14/Mar/17  Resolved: 25/Apr/16

Status: Closed
Project: Core Server
Component/s: Diagnostics, Querying
Affects Version/s: 3.3.3
Fix Version/s: 3.3.5

Type: Improvement Priority: Major - P3
Reporter: James Wahlin Assignee: James Wahlin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-23272 Remove misleading/not useful diagnost... Closed
Related
related to SERVER-23081 keyUpdates not set on MMAPv1 document... Closed
related to DOCS-7737 For 3.4: Add documentation for keysIn... Closed
is related to SERVER-23861 Add keysInserted & keysDeleted to ser... Backlog
Backwards Compatibility: Major Change
Sprint: Query 13 (04/22/16), Query 14 (05/13/16)
Participants:
Linked BF Score: 0

 Description   

Adding, updating and removing index entries can be an expensive part of any write operation, especially when a large # of index entries are involved. We should track this cost (at a minimum, a count of the # of changes performed) and record in both the profiler and in slow operation diagnostic logging.



 Comments   
Comment by Githook User [ 25/Apr/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-23271 Add MMAPv1 profiler test for 'nmoved'
Branch: master
https://github.com/mongodb/mongo/commit/506876412086623fdcc51dbec5dfcbe49c5361d7

Comment by Githook User [ 22/Apr/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-23271 Tests for keysInserted & keysDeleted
Branch: master
https://github.com/mongodb/mongo/commit/fe84406d22bec76ea93220272f1381220adea240

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-23271 Add keysInserted and keysDeleted metrics for CRUD ops
Branch: master
https://github.com/mongodb/mongo/commit/dc3d30af10b6859124d1ce8790ad8e097d1f06c7

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: Revert "SERVER-23271 Add keysInserted and keysDeleted metrics for CRUD ops"

This reverts commit 6bbaee174447ee1c9177c72bdd07f050ab07e901.
Branch: master
https://github.com/mongodb/mongo/commit/0bd67d53681e4826a44f2b45b136f50cd36de44b

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'benety', u'name': u'Benety Goh', u'email': u'benety@mongodb.com'}

Message: Revert "SERVER-23271 Fix dbtest for RecordStore::updateRecord contract change"

This reverts commit 2edcf2976ae45145af0c655d3e4a2dd9560b5b2f.
Branch: master
https://github.com/mongodb/mongo/commit/42415534834c82518dfdb4c75c3b22b75edb5eff

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-23271 Fix dbtest for RecordStore::updateRecord contract change
Branch: master
https://github.com/mongodb/mongo/commit/2edcf2976ae45145af0c655d3e4a2dd9560b5b2f

Comment by Githook User [ 21/Apr/16 ]

Author:

{u'username': u'jameswahlin', u'name': u'James Wahlin', u'email': u'james.wahlin@10gen.com'}

Message: SERVER-23271 Add keysInserted and keysDeleted metrics for CRUD ops
Branch: master
https://github.com/mongodb/mongo/commit/6bbaee174447ee1c9177c72bdd07f050ab07e901

Comment by James Wahlin [ 04/Apr/16 ]

After further discussion we are now leaning towards adding both a 'keysInserted' and a 'keysDeleted' metric, rather than a single 'keysAffected' metric. The motivation for this is added clarity on work done.

When an indexed value is changed in a document, this change is executed as a key removal + a key addition. With a single 'keysAffected' metric, an updated key value could be represented in one of 2 ways:
a) Both the insert and delete components on update have a value of 1 in the metric. So while an insert would increase 'keysAffected' by one, an update would increase by 2. This would keep this metric in-line with actual cost, but could be confusing to end users who would logically see an updated field as affecting one key.
b) An update has a value of 1 in the 'keysAffected' metric, on par with insert and delete operations. This would make any value easier to rationalize, but would not account for an updated value being equivalent (in work done) to a delete + insert.

Reporting 'keysInserted' and 'keysDeleted' helps clarify the actual work being performed and is easier to understand. We will have to educate users that an updated indexed value means 1 delete + 1 insert, but this seems easier to rationalize than the 'keysAffected' alternative.

Note: The above is meant to reflect insert and delete operations on non-multikey indexes. On update of an indexed array field, the values of keysInserted and keysDeleted could be larger than 1, reflecting the true number of index keys inserted and removed to satisfy the operation.

Comment by James Wahlin [ 25/Mar/16 ]

Our current plan is to add a "keysAffected" metric, that would be a sum of all index keys inserted, removed or updated during a given write operation.

Generated at Thu Feb 08 04:02:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.