[SERVER-49694] On a sharded cluster, nearest or hedged reads may not be routed to a near shard replica. Created: 17/Jul/20  Updated: 29/Oct/23  Resolved: 17/Aug/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.4.1, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Benjamin Caimano (Inactive) Assignee: Lamont Nelson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2020-07-17 at 13.09.39.png     File rsm_heartbeat.js    
Issue Links:
Backports
Documented
is documented by DOCS-13448 4.4 Known Issues Closed
Problem/Incident
Related
Backwards Compatibility: Minor Change
Operating System: ALL
Backport Requested:
v4.4
Steps To Reproduce:

I've attached a javascript test that logs the RTTs over the course of 20 pings. It shows 0 RTT reliably despite a failCommand fail point that should prevent that.

Participants:
Linked BF Score: 44

 Description   

 

Issue Status as of July 28, 2020

ISSUE SUMMARY

Due to an issue with reporting the round-trip time (RTT) of replica set members, on mongoS, "nearest" read preference and hedged reads do not determine the eligible nodes based on localThresholdMs.

USER IMPACT

On a sharded cluster, nearest reads or hedged reads are not necessarily routed to a near shard replica (that is, within 15ms round-trip time). These reads are instead dispatched to an arbitrary node, which can be more remote.

WORKAROUNDS

Installations with distributed sharded clusters that rely on read preference "nearest" should not upgrade to 4.4.0. Instead, upgrade to 4.4.1.

AFFECTED VERSIONS

This affects version 4.4.0.

FIX VERSION

This issue is corrected in MongoDB 4.4.1.

 

Original Description

When looking a substantial RTT for non-colocated hosts, I'm seeing clean .8 factor decreases on ping intervals which trends towards 0. I see that the factor is .2 from here. My conclusion is we're passing in RTT=0 for each update after the first.



 Comments   
Comment by Githook User [ 12/Nov/20 ]

Author:

{'name': 'jeff-allen-mongo', 'email': 'jeffrey.allen@10gen.com', 'username': 'jeff-allen-mongo'}

Message: (DOCSP-11857): SERVER-49694 fixed in 4.1.1
Branch: master
https://github.com/mongodb/docs/commit/04cdee0b0ef9df1931bbeea98b5c48af6906afa6

Comment by Githook User [ 20/Aug/20 ]

Author:

{'name': 'jeff-allen-mongo', 'email': 'jeffrey.allen@10gen.com', 'username': 'jeff-allen-mongo'}

Message: (DOCSP-11857): SERVER-49694 fixed in 4.1.1
Branch: v4.4.1
https://github.com/mongodb/docs/commit/b976b3239681439dc3fa4d504b58b34ab2a0c593

Comment by Jeffrey Allen [ 17/Aug/20 ]

Hi lamont.nelson, we'd been documenting this ticket as a Known Issue in 4.4. Can we now remove this ticket from that section?

Comment by Githook User [ 17/Aug/20 ]

Author:

{'name': 'LaMont Nelson', 'email': 'lamont.nelson@mongodb.com', 'username': 'lamontnelson'}

Message: SERVER-49694: fix latency measurement in RSM; change latency measurement for command responses to Microseconds

(cherry picked from commit d09a0da1f4ccca9b087ba21bba66c88e0abd07d0)
Branch: v4.4
https://github.com/mongodb/mongo/commit/434d078caab5ab814d3c40db9418844311fc4eac

Comment by Githook User [ 14/Aug/20 ]

Author:

{'name': 'LaMont Nelson', 'email': 'lamont.nelson@mongodb.com', 'username': 'lamontnelson'}

Message: SERVER-49694: fix latency measurement in RSM; change latency measurement for command responses to Microseconds
Branch: master
https://github.com/mongodb/mongo/commit/d09a0da1f4ccca9b087ba21bba66c88e0abd07d0

Comment by Githook User [ 13/Aug/20 ]

Author:

{'name': 'LaMont Nelson', 'email': 'lamont.nelson@mongodb.com', 'username': 'lamontnelson'}

Message: Revert "SERVER-49694: fix latency measurement in RSM; change latency measurement for command responses to Microseconds"

This reverts commit b2c1fa4f121fdb6cdffa924b802271d68c3367a3.
Branch: master
https://github.com/mongodb/mongo/commit/40ca8fbe45bb723f08261c4f6476fd79b6f276b0

Comment by Githook User [ 05/Aug/20 ]

Author:

{'name': 'LaMont Nelson', 'email': 'lamont.nelson@mongodb.com', 'username': 'lamontnelson'}

Message: SERVER-49694: fix latency measurement in RSM; change latency measurement for command responses to Microseconds
Branch: master
https://github.com/mongodb/mongo/commit/b2c1fa4f121fdb6cdffa924b802271d68c3367a3

Comment by Lamont Nelson [ 23/Jul/20 ]

Yes, Microseconds makes sense. I've started changing the code and it seems this will impact a number of files as I move through the call sites and their dependencies. I'm currently changing the types that use this field for consistency, and to preserve information.

Alternatively, I could do a BFS and try to keep the changes to one level deep, but it seems that being consistent in this regard is the right thing to do since this would save any users of RemoteCommandResponse from having to instantiate their own timer (like the old RSM) if they care about the latency.  

If anyone feels strongly either way let me know.

Comment by Andy Schwerin [ 22/Jul/20 ]

So, Microseconds? That's the natural return type Timer::elapsed().

Comment by Lamont Nelson [ 22/Jul/20 ]

This will require changing this type to a sub-millisecond type, so that the code here will have a nonzero value.

Generated at Thu Feb 08 05:20:33 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.