[SERVER-65289] Improvements to Mirrored Reads Metrics and Visibility Created: 06/Apr/22  Updated: 09/Nov/23  Resolved: 27/Jun/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.6, 4.4.13
Fix Version/s: 6.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Diego Rodriguez (Inactive) Assignee: Jason Chan
Resolution: Fixed Votes: 0
Labels: 6.1-targeted, diagnosibility, troubleshooting
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File mirrored_reads_metrics.png    
Issue Links:
Documented
is documented by DOCS-15700 Investigate changes in SERVER-65289: ... Closed
Related
is related to SERVER-70103 Fix the accuracy of the test assertio... Closed
Backwards Compatibility: Fully Compatible
Sprint: Service Arch 2022-06-13, Service Arch 2022-06-27, Service Arch 2022-07-11
Participants:
Case:

 Description   

Hi Team,

Starting in v4.4 we introduced the Mirrored Reads feature, and we have a couple of server metrics tracking that:

  • ss mirroredReads seen: tracks the number of operations that can be mirrored ("mirrorable").
  • ss mirroredReads sent: tracks the numbe of "mirrorable" operations that were actually sent to Secondaries.

The problem is that there is no indication in the Secondaries' Diagnostic Data or Logs of the amount of reads executed as a result of the Mirror Reads feature.

For Troubleshooting purposes, it would be nice to have the following:

  1. When an instance is a Secondary and it receives mirrored reads, track them in the Diagnostic Data with a metric similar to ss mirroredReads executed.
  2. When an instance is a Secondary and it eventually logs a "slow" (as per slowms threshold) mirrored read, add a field in the logs to clearly show the origin of this call like "mirrored: true"

These changes should be very helpful when troubleshooting, and the suggested changes should be added to both v4.4 and v5.0.



 Comments   
Comment by Alex Li [ 24/Oct/22 ]

The mirroredReads.received metric is now changed to mirroredReads.processedAsSecondary to provide a more accurate name.

Comment by Jason Chan [ 27/Jun/22 ]

This patch adds the following:

  • serverStatus now outputs a mirroredReads.received field that secondaries increment. This represents the number of mirrored operations received by a secondary.
  • On slow operation logs, secondaries will append a mirrored: true attribute to the logs to indicate that the operation was a mirrored operation.
Comment by Githook User [ 24/Jun/22 ]

Author:

{'name': 'Jason Chan', 'email': 'jason.chan@mongodb.com', 'username': 'jasonjhchan'}

Message: SERVER-65289 Improvements to Mirrored Reads Metrics and Visibility
Branch: master
https://github.com/mongodb/mongo/commit/e4e6a561a4579c8a11c964f0a12cfb6391bc9d34

Generated at Thu Feb 08 06:02:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.